Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 37 ยท Metrics and Frameworks

Alert Tuning and Detection Quality

Learn to tune SIEM alerts safely: measure precision and recall, find noisy rules, add exclusions, thresholds and context, and test changes against known attacks.

Every SOC reaches the same point: the rules produce more alerts than people can read. Analysts start skimming, and the one alert that mattered sits in the queue unopened. Alert tuning is the practice of improving detections so that they fire on what matters and stay quiet on what does not. The danger is that tuning, done carelessly, removes real detections along with the noise. This lesson teaches the measurements and safe techniques.

Why alerts are noisy

  • Rules written against generic behavior (for example "PowerShell ran") that is normal on admin machines.
  • Missing context, so the rule cannot tell a scanner from an attacker.
  • Thresholds set by guesswork instead of data.
  • Duplicates: five rules and three tools all alerting about one event.
  • Changes in the environment (new software, new scanners) after the rule was written.

Measuring quality: precision and recall

Two ideas from every confusion matrix apply here.

  • Precision = true positives / all alerts. "When this rule fires, how often is it right?" Low precision means noise.
  • Recall (detection rate) = true positives / all real attacks. "Of the real attacks, how many did it catch?" Low recall means blind spots.

Tuning for precision alone drives recall down: a rule that never fires has perfect precision and catches nothing. Always check both.

Two numbers to balance

Precision How many alerts are real?
Recall How many attacks do we catch?

Raising one often lowers the other. Check both after every change.

1 / 2
Python Precision and recall of a rule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Version 2 looks wonderful if you count alerts, but it quietly misses nine real attacks: excluding all administrators removes exactly the accounts attackers love. Version 3 keeps recall high while cutting alerts by roughly 90 percent.

Tuning techniques

Find the noisy rules first

Rank rules by alert volume and by false positive rate. A handful of rules usually produce most of the noise, so fix those first.

Python Rank rules by noise

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One rule is responsible for about 70 percent of all false positives: that is where tuning effort pays most.

Add context instead of removing the rule

Better than deleting a noisy rule is making it smarter. "Admin tool used" can require that the user is not in the IT admins group, or that the host is not an admin workstation, or that the tool ran from an unusual parent process. Enrichment from your asset and identity tables makes this possible.

Use thresholds from data

Do not pick "5 failed logins in 1 minute" from thin air. Look at the real distribution: what is the 99th percentile of failed logins per user per minute on normal days? Set the threshold above normal but below attack volume, and revisit it as behavior changes.

Write narrow, owned, expiring exclusions

An exclusion (allow-list entry) should be as specific as possible (this scanner at this address, this account on this host), have a named owner and reason, and an expiry or review date. "Ignore everything from the 10.0.0.0/8 network" is not tuning; it is switching the detection off.

Correlate weak signals

Several low-value signals together may deserve an alert even though none does alone: a rare process plus a new outbound destination plus a failed-then-successful login on the same host within an hour.

Aggregate and suppress duplicates

Group repeated alerts for the same entity into one case, and suppress repeats for a short window. Keep the count, so that a burst is still visible.

Python Suppress duplicate alerts for 10 minutes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Five raw alerts become three cases, with counts that keep a repeated infection visible.

A safe tuning workflow

  1. Measure the current precision, recall (if known) and volume.
  2. Understand why the false positives happen: read a sample, find the pattern.
  3. Change one thing at a time, on a test copy or in a report-only mode.
  4. Back-test against historical logs, including the logs of past real incidents. The rule must still catch them.
  5. Review with another engineer, record the change and the reason in version control.
  6. Deploy and watch for a few weeks, ready to roll back.
  7. Re-check quarterly; environments drift.

Never tune by deleting alerts that "feel" like noise without reading the cases. The scariest attack of the year may look exactly like a hundred harmless events, hidden in the noise you removed.

Detection-as-code

Treat rules like software: keep them in a repository, review changes, test them with sample logs (a "good" log must not alert and a "bad" log must), and deploy through a pipeline. Rule metadata should include the ATT&CK technique (coverage), data source, owner, severity, expected volume, and a runbook link.

Common mistakes

  • Optimizing for fewer alerts rather than better ones.
  • Broad exclusions that nobody reviews.
  • Skipping back-testing against real attack logs.
  • No owner for rules, so nobody maintains them.
  • Tuning one rule while a duplicate rule in another tool keeps alerting.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does low precision indicate about a detection rule?

    Precision is the share of alerts that are real; low precision means noise.
  2. 2. A tuned rule excludes all administrator accounts and alerts drop 90%. What should you check next?

    Attackers target privileged accounts, so a broad exclusion can destroy detection quality.
  3. 3. Which exclusion is the best practice?

    Good exclusions are specific, owned, justified and reviewed.
  4. 4. Why back-test a tuned rule against past incident logs?

    Past incident data is a ready-made test set for recall.

Summary

  • Measure precision and recall, and find the few rules that produce most of the noise.
  • Prefer adding context, data-based thresholds, correlation and deduplication to deleting detections.
  • Keep exclusions narrow, owned and expiring; back-test every change against known attacks.
  • Treat rules as code: version, review, test and re-check regularly.

Next: Project: A Brute-Force Detection Pipeline