Skip to content
Tutorials Center
My learning

Topics

Lesson 174 of 182 ยท Projects

Project: Log Analyzer

Build a log analyzer in C# in four stages: a regex line parser, LINQ reports on thousands of entries, a zero-copy span parser, and a streaming pipeline for large and gzipped files.

Every server writes logs, and when something goes wrong somebody has to read them. A log file can have millions of lines, and the question is rarely "what is on line 5,000,000?" but "how many errors were there per service last night, what was the slowest request, and did it start at the same minute as the deploy?". Answering such questions is the daily bread of operations engineers, and it is a rich exercise in everyday C#: text parsing with regular expressions, summarizing with LINQ, speeding up with spans, and handling files too big for memory with streaming.

You build the analyzer in four stages, and every stage is a complete program. In stage 1 you parse a single log line with a regex and learn what to do with lines that do not fit. In stage 2 you parse five thousand generated log records and answer questions with LINQ: counts by level and service, error spikes over time, the most common errors and latency percentiles. In stage 3 you write a second parser with spans that does the same work without allocating a string for every piece, and prove that both agree. In stage 4 you stream large files, including gzip-compressed ones, in constant memory with an aggregator that can be merged, so the files can be processed in parallel.

Because the sandbox has no real log files, a small generator creates realistic ones from a fixed random seed, so every run produces identical lines and identical results. The generated log contains the awkward things real logs contain: stack-trace continuation lines, a few corrupted lines and a long tail of slow requests. In your own project you would simply point the analyzer at your files.

The log format

The log is a text file with one entry per line:

TEXT
2026-10-21 08:15:03.120 [WARN] checkout - Payment retry 2/3 for order 1007
2026-10-21 08:15:06.950 [ERROR] checkout - Payment failed for order 1007: gateway timeout
    at Shop.Payments.Gateway.Charge()
    at Shop.Checkout.Handler.Run()
2026-10-21 08:15:07.001 [INFO] checkout - Order 1008 placed, total 59.90 EUR (took 1840 ms)

That is a timestamp (UTC, millisecond precision), a level in brackets, the name of the service, a dash, and the message. Lines that start with whitespace are continuation lines (a stack trace belongs to the entry above it). Anything else that doesn't match is malformed. A robust analyzer never crashes on a bad line: it counts it and moves on.

Stage 1: parsing one line with a regular expression

A regular expression describes the shape of a line. With named groups ((?<level>...)) you read each part by name instead of by position, so the pattern can change without breaking the code that uses it. Three habits make a log parser trustworthy:

  1. Anchor the pattern with ^ and $ so it must match the whole line, not just a piece of it.
  2. Validate after matching. The pattern \d{4}-\d{2}-\d{2} accepts 2026-13-45, which is not a date. The regex tells you the shape; DateTime.TryParseExact and Enum.TryParse tell you whether the values are real.
  3. Use TryParse style, not exceptions. A corrupted line is expected in a log, so the parser returns false, and the caller decides what to do. Throwing and catching exceptions for every bad line would also be slow on a large file.

Also pass a match timeout to the Regex constructor. Logs can contain text that an attacker controls (a user agent, a user name), and a badly written pattern can take exponential time on crafted input. A timeout turns "hang the whole analyzer" into an exception you can handle (regular expressions).

The ExplicitCapture option means only named groups are captured, so ordinary parentheses are free for grouping and cost nothing. We write the pattern as a raw string literal ("""..."""), where backslashes need no doubling (raw strings).

C# Stage 1: a regex parser with validation and TryParse

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

What the output shows:

  • Three outcomes for a line, not two. A line is an entry, a continuation (it starts with whitespace) or malformed. The empty string and ### corrupted line ### are malformed. A parser that only knows "success" and "failure" would count stack traces as corruption.
  • The impossible date and the unknown level pass the regex but fail validation. 2026-13-45 99:99:99.999 has the right shape; DateTime.TryParseExact rejects it. FATAL has the right shape for a level but is not in our enum. Both are reported as malformed. This is the division of labour: regex for structure, TryParse for meaning.
  • [NotNullWhen(true)] out LogEntry? entry tells the compiler (and therefore callers) that entry is not null when the method returns true, so after if (TryParse(line, out var entry)) you can use entry without a null check (nullable reference types).
  • Normalizing replaces every number with #. Payment failed for order 1007: gateway timeout and the same message for order 2214 become one identical string, which is what lets stage 2 count "the same error" even though the order ids differ. Group on the normalized message, display an example of the original.
  • Static readonly regexes. Each pattern is built once and reused. Creating a Regex inside the method that uses it would re-parse the pattern for each call. (In an ordinary project you would write [GeneratedRegex(...)] on a partial method to have the compiler generate the matching code; the lesson editor has no source generators, so the plain fields are used here.)

Regexes are very good at recognizing a known format and very bad at being a general parser. If the format has quoting, nesting or escaped separators (CSV with commas inside quotes, JSON), use a real parser. If a log format is stable and simple, as here, a regex is right.

Stage 2: answering questions with LINQ

Now scale up. A parser that handles one line is useful; the real value is in summaries. This stage generates 5,000 log records (5,344 lines once stack traces and corrupted lines are counted), parses them all, and answers six questions with LINQ. The reader function treats a continuation line as belonging to the previous entry (it adds to a counter on that entry), counts malformed lines, and produces a list of LogEntry.

The questions and the operators that answer them (LINQ operators):

Question Operators
How many entries per level? CountBy (new in .NET 9)
Which service produces which levels? GroupBy, then a second CountBy inside each group
When do errors spike? bucket the time with arithmetic on Ticks, CountBy, OrderBy
What are the most common errors? normalize with the regex, CountBy, OrderByDescending, Take
How slow are requests? Select + Where to pull out the milliseconds, sort, pick percentiles
Which requests were slowest? OrderByDescending, Take

Percentiles deserve a sentence. The median (p50) is the value in the middle; p95 is the value that 95% of requests were faster than. Averages hide the long tail that users actually feel, so latency reports use percentiles. The simple "nearest rank" method used here sorts the values and takes the one at index ceil(p/100 * n) - 1.

C# Stage 2: reports over 5,000 entries with LINQ

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Read the report with these ideas in mind:

  • CountBy replaces a very common pattern. Before .NET 9 you wrote GroupBy(key).Select(g => (g.Key, g.Count())), which builds a whole group of items to count them. CountBy counts in one pass without keeping the items (LINQ operators). ToDictionary() on the key/value pairs turns a result into a lookup for the table.
  • Time buckets by arithmetic. ticks - ticks % bucket rounds a time down to the start of its 10-minute window, in one line, with no string formatting. Use the same trick for hours or days.
  • Group on the normalized text. If you grouped errors on the raw message, every "Payment failed for order N" would be unique and the top-5 would be a list of ones. Normalizing is what produces counts that mean something.
  • OfType<int>() on int? values keeps the non-null ones and unwraps them, a compact way to drop the entries that have no "took" value (null operators).
  • Sorting is stable and total. Every OrderByDescending that could tie is followed by ThenBy on something unique (the key or the time). Without it the order of equal items would depend on the input order; for a report that is compared against a previous run, make the order fully determined.
  • The latency histogram has a long tail. The mean is far above the median because a few very slow requests pull it up, and the p99 is many times the p50. This is why you report percentiles and not just an average.
  • Everything is a pass over a list in memory. Each Where or CountBy walks all entries again. For 5,000 entries that costs nothing. For 50 million entries it would be slow, and the list itself would not fit in memory; stage 4 addresses both.

When you build reports from LINQ, enumerate each query once and keep the result with ToList() or ToDictionary() when you will use it again. A query like entries.Where(...) stored in a variable runs again every time you enumerate it, and that is a classic hidden cost (deferred execution).

Stage 3: a zero-copy span parser

Look at what the regex parser does for each line: it creates a Match object, a GroupCollection, one Group per named group, and a new string for each .Value (time, level, service, message). Of those allocations only two are really needed: the service and the message that we store. For a few thousand lines nobody cares. For billions of lines the garbage collector spends its time cleaning up strings that existed for a microsecond.

Spans fix that. A ReadOnlySpan<char> is a view onto a piece of memory (here, onto the characters of the line) with no copy. Slicing it with line[..23] or line[25..] costs nothing: it just adjusts a pointer and a length. The standard library accepts spans almost everywhere: DateTime.TryParseExact(ReadOnlySpan<char>, ...), Enum.TryParse(ReadOnlySpan<char>, ...), int.Parse(ReadOnlySpan<char>). So we can parse the whole line without creating a single intermediate string, and make strings only for the two fields we keep (spans and memory).

Even splitting a big text into lines can avoid allocation. text.AsSpan().EnumerateLines() yields each line as a span of the original string, instead of text.Split('\n'), which allocates an array and a string for each line.

The price of this speed is that you write the parsing by hand: no pattern describes the format any more, so you check each position yourself. That is more code and more room for off-by-one mistakes. So we do two things professionals do: keep the regex parser as the reference implementation, and compare the two on thousands of lines, including the corrupted ones. If the fast parser disagrees with the simple one anywhere, we have a bug.

The program measures allocations with GC.GetAllocatedBytesForCurrentThread(). It does not print byte counts (they vary between runtime versions) and instead checks that the span version allocates less than half as much.

C# Stage 3: a span parser checked against the regex parser

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The comparison section is the part to copy into your own projects:

  • Two implementations, one contract. The regex parser is slow but obviously right; the span parser is fast but hand-written. A loop that runs both over real input and demands identical answers is the cheapest correctness test you can write. Add the awkward cases you can think of (the second loop) and every future bug becomes one more line there.
  • Why the span parser checks IsUpperAscii and IsLowerAscii. The regex says [A-Z]+ and [a-z]+. Without the same restriction the span parser would accept [info] (ignoreCase: true) or a service name with digits, and the comparison would show a DISAGREE. The first version of such a parser nearly always has one. Use the output of the awkward-input loop to see how each case is decided.
  • ReadOnlySpan<char> is a ref struct. It can live only on the stack: you cannot store it in a field of a class, capture it in a lambda, or keep it across an await. That restriction is what makes it safe: a span can never outlive the memory it points to. When you need to keep data, call .ToString() (as we do for the two fields we store).
  • Slicing is free, .ToString() is not. rest[..separator] creates no string; ToString() allocates one. The rule of thumb for fast text code: slice, compare and parse with spans, and allocate only for what you will keep.
  • Do not guess performance; measure. The allocation check uses a counter that the runtime provides, which is stable, unlike a stopwatch on a shared machine. For timing, the standard tool is the BenchmarkDotNet NuGet package, which handles warm-up, statistics and noise (not available in this editor; see performance basics).
  • EnumerateLines() against Split('\n'). Both find the same entries. The first yields each line as a span of the original string; the second allocates an array and every line string up front. On a 100 MB log the difference is hundreds of megabytes of garbage.

Spans are a performance tool, not a style. Most code should use plain strings and regexes, which are clear and fast enough. Reach for spans when a profiler shows that text processing and allocation dominate, as in a parser that runs on every line of a huge file.

Stage 4: streaming big files

All the code so far reads the log fully into memory (ToList(), string.Join). That works until the file is larger than your RAM. The fix is to stream: read one line at a time, update a few counters, and forget the line. Memory use is then independent of the file size, which is the difference between an analyzer that handles a 50 MB file and one that handles 50 GB.

That changes the shape of the program. You can no longer sort all the latencies to find a percentile, because you never hold them. Streaming code uses aggregates that can be updated one item at a time:

  • Counters in dictionaries (per level, per service, per normalized error). The number of distinct keys is small even when the number of lines is huge.
  • A histogram for latency: an array of counts for buckets of 25 ms. Each request adds one to a bucket. The p95 is then "the bucket in which the 95th percentile falls" โ€” an approximation within one bucket width, using constant memory.
  • A Merge method that adds another aggregate into this one. Merging works because counting is associative: counts from two halves of the data add up to the counts of the whole. That lets you analyze several files in parallel and combine the results at the end (this is the idea behind MapReduce).

The file handling uses two more standard tools. StreamReader.ReadLineAsync(CancellationToken) reads without blocking a thread and honours cancellation. And because streams compose, reading a .gz file is only one more layer: wrap the file stream in a GZipStream and everything above it, the StreamReader and our parser, stays the same (streams and I/O).

The program writes two log files (one plain, one gzip-compressed, 6,000 log records each, 6,400 lines with the stack traces) to the current directory, analyzes them concurrently, merges the results, prints the report, and cross-checks the streaming numbers against an exact in-memory calculation. A final check shows that a cancelled token stops the analysis.

C# Stage 4: streaming, gzip, merging and a final report

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

How the streaming design works:

  • Memory is bounded by the number of distinct keys, not lines. Nothing in LogStats grows with the file, except the three dictionaries, and they hold levels (4), services (4) and normalized error messages (a handful). The normalization from stage 1 is what keeps ErrorKinds small: without it every order number would be its own key and the dictionary would grow with the file.
  • Merge makes parallelism trivial. Each file is analyzed by its own LogStats on its own task, with no sharing, and therefore no locks. Only after Task.WhenAll do we combine them on one thread. "Shared nothing, then merge" is the simplest correct way to parallelize aggregations (task combinators).
  • The histogram trades exactness for constant memory. The report prints p95 <= N ms because only the bucket is known. The cross-check proves the approximation is within one bucket width of the exact answer. If you need exact percentiles on huge data, look at streaming sketch algorithms (t-digest, HDR histograms); the architecture remains the same.
  • await using and using var dispose the stream chain in the right order even when an exception or cancellation interrupts reading (disposal).
  • The useAsync: true flag on FileStream makes reads truly asynchronous on the operating system level; without it ReadLineAsync would still work but would occupy a thread-pool thread per file while waiting for the disk.
  • Cancellation is cooperative. The token passed to ReadLineAsync makes it throw OperationCanceledException as soon as it is observed. We pass an already-cancelled token to show the effect; in a real tool the token would be wired to Ctrl+C (cancellation).
  • Gzip is just a layer. Switching on the file extension to wrap the stream means the analysis code does not know or care that the data was compressed. The same trick works for BrotliStream, ZipArchive entries, a network stream or standard input.
  • The final report is a StringBuilder and is returned as a string, not printed inside the method. That makes it testable and lets the caller write it to the console, a file or an HTTP response.

Never decide that a log line is "malformed" and throw it away silently. The Malformed counter is part of the report for a reason: if it jumps from 60 to 60,000 after a deploy, the log format changed and every other number in the report is suspect. An analyzer should be loud about its own blind spots.

Where to take it next

  1. Command-line options. Accept --level=ERROR, --service=checkout, --since=08:30 and a list of files, and filter before aggregating. In a normal project System.CommandLine parses the arguments; with args alone you can use list patterns, as in the text adventure parser.
  2. Follow mode. Like tail -f: keep a file open, read new lines as they are written and update a live view. Use a FileSystemWatcher or poll with a delay and a cancellation token.
  3. A session detector. Group entries by user id (extract it with a named group) and report users with five failed logins in a minute: a sliding window over time-ordered entries.
  4. JSON logs. Real systems often write one JSON object per line. Replace the parser with System.Text.Json's Utf8JsonReader, which, like spans, avoids allocations. The aggregator stays exactly the same (JSON).
  5. Channels. Split reading and aggregating into a producer and consumers connected by a bounded Channel<string[]> of batches, so disk reads and CPU work overlap (channels).
  6. An HTTP endpoint. Serve the report from a minimal API and refresh it when new files appear.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does the parser run DateTime.TryParseExact and Enum.TryParse after the regex matched?

    A pattern such as \d{2} accepts 13 as a month. The regex validates structure, and the parse methods validate meaning. Together they make sure only real timestamps and known levels become entries.
  2. 2. What is the main advantage of ReadOnlySpan<char> slices such as line[..23] over line.Substring(0, 23)?

    A span is a pointer and a length into existing memory. Slicing adjusts those two values and allocates nothing, while Substring creates a new string each time.
  3. 3. Why can't ReadOnlySpan<char> be stored in a field of a normal class or kept across an await?

    ref structs are restricted to the stack. That restriction guarantees a span cannot be used after the underlying memory is gone, which is what makes spans safe.
  4. 4. Why does the streaming analyzer report percentiles from a histogram instead of sorting all latencies?

    Streaming means never holding the whole data. Bucketed counts use constant memory and can be merged from parallel workers; the price is that the result is only accurate to the bucket width.
  5. 5. Which property of the aggregates makes it safe to analyze two files in parallel and combine them afterwards?

    With no shared state there is nothing to lock. Counts and histogram buckets simply add up, so merging the per-file results gives the same totals as one pass over all lines.

Summary

  • A regex describes the shape of a log line; named groups make fields readable. Anchor patterns, give them a match timeout, validate values after matching, and use TryParse so corrupted lines are a counted outcome and not an exception.
  • Distinguish three kinds of lines: entries, continuation lines (stack traces) and malformed lines; report all three.
  • Normalize messages (replace numbers) before grouping, so the same problem counts as one.
  • Summarize with LINQ: CountBy, GroupBy, time buckets by arithmetic on ticks, percentiles by sorting. Make report ordering fully deterministic with ThenBy.
  • Spans (ReadOnlySpan<char>, EnumerateLines) parse text without allocating strings; keep a simple reference implementation and compare both on real and awkward input.
  • To handle files bigger than memory, stream: read line by line (StreamReader.ReadLineAsync), compose layers such as GZipStream, and keep only mergeable aggregates (counters, histograms).
  • "Shared nothing, then merge" parallelizes aggregation without locks; cancellation tokens make long runs stoppable.

Next: Project: Background Job Queue