Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 34 ยท Projects

Project: Log Analyzer

Build a log analyzer in PowerShell: parse lines with a named-group regex, count by level, find the top repeated errors and return one summary object. Python stand-in included.

Logs are the diary of every application, and they are usually too long to read. A log analyzer turns thousands of lines into a few answers: how many errors, which ones repeat, and when did trouble start. This project combines regular expressions, the pipeline, grouping and custom objects into a small, useful tool, and it is the kind of script people really keep in their toolbox.

The real analyzer reads a file with Get-Content, which needs a file system, so it is shown as read-only PowerShell. The runnable stand-ins are written in Python, which runs in your browser, and they work on a sample log kept in the code. They follow the same steps, so you can compare each line with its PowerShell counterpart. They are models of the logic, not PowerShell.

The log format

Our sample log has one event per line: a timestamp, a level, then a free-text message.

TEXT
2026-03-02 08:15:42 INFO Service started on port 8080
2026-03-02 08:16:07 WARN Slow query took 2300 ms (orders-db)
2026-03-02 08:17:30 ERROR Database timeout after 30 s (orders-db)

Real logs are messy. Some lines are continuation lines of a stack trace, some are blank, and some have a different shape. A solid analyzer parses what it understands, and counts what it does not instead of crashing or silently dropping it.

From lines to answers

Read Get-Content, one line at a time.
Parse A regex with named groups.
Group Count per level and per message.
Summarize Return one object with the facts.

Parse once, then every question is a cheap query.

1 / 2

Step 1: parse each line into an object

A regular expression with named groups pulls the three parts out. In PowerShell, -match fills the automatic $Matches hashtable, where each group is a key (see Regular Expressions and Text Processing):

POWERSHELL
param([Parameter(Mandatory)][string]$LogPath, [int]$Top = 3)

$pattern = '^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?<level>[A-Z]+) (?<message>.+)$'
$unparsed = [System.Collections.Generic.List[string]]::new()

$entries = Get-Content -LiteralPath $LogPath | ForEach-Object {
    if ($_ -match $pattern) {
        [pscustomobject]@{
            Time    = [datetime]$Matches.time
            Level   = $Matches.level
            Message = $Matches.message
        }
    }
    elseif ($_.Trim()) {
        $unparsed.Add($_)          # keep what we could not understand
    }
}

The [pscustomobject] lines become objects in the output stream, so $entries collects them. Lines that do not match are not output; they are added to a list, so you can report how many were skipped. We use the Add method of a generic list, because it is cheap, while += on an array copies the whole array every time (see Arrays and Hashtables). If the log might produce zero or one entry, wrap the pipeline in @( ... ) so $entries is always an array.

Step 2: count by level

POWERSHELL
$byLevel = $entries | Group-Object -Property Level -NoElement |
    Sort-Object -Property Count -Descending
$byLevel | Format-Table Name, Count

-NoElement keeps only the key and the count, which saves memory on large logs. For a quick summary you could stop here.

Step 3: the top repeated errors

Messages contain changing numbers (30 s, 2300 ms, user ids), so exact comparison would see every error as unique. The trick is to normalize them: replace digit runs with a placeholder, and then group:

POWERSHELL
$topErrors = $entries |
    Where-Object { $_.Level -eq 'ERROR' } |
    Group-Object { $_.Message -replace '\d+', 'N' } |
    Sort-Object -Property Count -Descending |
    Select-Object -First $Top -Property Count, @{ Name = 'Message'; Expression = { $_.Name } }

Now "Database timeout after 30 s" and "Database timeout after 45 s" become one pattern, Database timeout after N s, with a count of two. Choosing what to normalize is a design decision: numbers, hex ids and file paths are common candidates.

Step 4: return one summary object

A script should output an object, not formatted text, so that another script can use it. Build the summary with [pscustomobject]:

POWERSHELL
$sorted = $entries | Sort-Object -Property Time

[pscustomobject]@{
    TotalLines  = $entries.Count + $unparsed.Count
    Parsed      = $entries.Count
    Unparsed    = $unparsed.Count
    First       = $sorted[0].Time
    Last        = $sorted[-1].Time
    ByLevel     = $byLevel | ForEach-Object { '{0}={1}' -f $_.Name, $_.Count }
    TopErrors   = $topErrors
}

Run the script and you see a tidy list. Pipe it into ConvertTo-Json -Depth 4 for a machine-readable report, or into Format-List for people (see Formatting and Output). Because the object keeps real types ([datetime], numbers), any follow-up question is easy: $summary.Last - $summary.First gives a TimeSpan.

The stand-in, part one: parse and count

Python's re module has the same named-group idea, written (?P<name>...). The sample log is a multi-line string, split into lines like Get-Content would return. This first demo parses the lines, keeps count of the ones it does not understand, and counts per level.

Python Stand-in: parse lines and count levels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The indented stack-trace line did not match the pattern, so it landed in unparsed instead of vanishing. In PowerShell the same happens in the elseif branch.

The stand-in, part two: top errors and the summary

The second demo repeats the parsing, then normalizes messages with re.sub (PowerShell's -replace), groups the errors, and builds the summary. A Python dict plays the role of the [pscustomobject], and json.dumps of ConvertTo-Json.

Python Stand-in: top errors and a summary object

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that three "Payment declined" lines collapsed into one pattern with a count of three, even though each had a different order number. That is exactly what the -replace '\d+', 'N' normalization does in the PowerShell script, and it is what makes the "top errors" list meaningful.

Improving the analyzer

  • Time windows. Group by hour with Group-Object { $_.Time.ToString('yyyy-MM-dd HH') } to see when errors peak.
  • Large files. Get-Content sends lines down the pipeline one at a time, but storing every entry in $entries keeps them all in memory. For huge files, use Select-String to pre-filter (or -ReadCount 1000 for speed, after which each pipeline item is an array of lines), or keep only running counts if counts are all you need.
  • Different formats. Keep the pattern in a parameter, or detect JSON log lines and use ConvertFrom-Json for those (see CSV, JSON and XML).
  • Parameters. Add -Level and -Since filters with [ValidateSet('DEBUG','INFO','WARN','ERROR')].
  • Tests. The parse step is a perfect unit to test: feed it a few lines and assert on the object, as in Testing and Style with Pester.
  • Alerts. If ERROR counts exceed a limit, write to the event log or send a message.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the analyzer replace digit runs with N before grouping error messages?

    Without normalization every message with a different number is unique, and the top-errors list would show every line with a count of one.
  2. 2. In PowerShell, what fills the $Matches hashtable?

    After a successful -match, $Matches holds the whole match at key 0 and each named group under its name, such as $Matches.level.
  3. 3. Why should a script return a [pscustomobject] summary rather than formatted text?

    An object keeps real types, so callers can format it, export it to JSON or CSV, or compute with it. Text can only be read.
  4. 4. A stack-trace line does not match the log pattern. What should the analyzer do?

    Real logs contain lines you do not expect. Counting them keeps the analyzer robust and tells you how complete the parse was.

Summary

  • Parse each line once with a named-group regex into an object; keep what does not match and count it.
  • Group-Object with -NoElement counts per level; a script-block key groups by a normalized message.
  • Normalizing numbers with -replace '\d+', 'N' turns many unique errors into a few patterns.
  • Return one [pscustomobject] summary with real types, so the result can be formatted, exported or tested.
  • The Python demos run the same steps on sample data; the PowerShell script itself is shown read-only.

Next: Project: File Organizer.