Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 80 ยท Network Security

Intrusion Detection and Network Monitoring

Learn how IDS and IPS work, how signature and anomaly detection differ, what flow logs show, and build a simple port-scan detector in Python.

Firewalls decide what may pass, but they cannot tell you that something suspicious is happening inside the traffic they allow. An attacker who guesses a password on an open SSH port or probes your servers one port at a time may never trip a single block rule. To notice this activity you need monitoring: watching what the network is doing and raising an alert when it looks wrong.

In this lesson you'll learn what an intrusion detection system (IDS) and an intrusion prevention system (IPS) do, how signature-based and anomaly-based detection differ, what network flow logs contain, and how to write a small port-scan detector. All data in the demos is generated, so nothing touches a real network. For the attacks you'll be detecting, see Common Network Attacks and Defenses.

IDS versus IPS

An intrusion detection system (IDS) watches a copy of the traffic and alerts. An intrusion prevention system (IPS) sits directly in the path of the traffic and can also block it. The detection logic can be identical; the difference is where the box is placed and what it is allowed to do.

  • Network-based (NIDS/NIPS): inspects packets or flows on a network segment. Good for seeing scans, exploit attempts and strange protocols across many hosts.
  • Host-based (HIDS): runs on one machine and watches local things: log files, file changes, new processes, login events. It sees what the network cannot, such as activity inside encrypted traffic after decryption.
flowchart LR
    NET["Internet"] --> FW["Firewall"]
    FW --> SW["Switch"]
    SW --> SRV["Servers"]
    SW -. "mirrored copy (SPAN/TAP)" .-> IDS["IDS sensor"]
    IDS --> SIEM["Alerts to SIEM<br/>and analysts"]

An IDS usually watches a mirrored copy of traffic, so it cannot slow the network down. An IPS would sit inline instead.

An inline IPS has a cost: if it is wrong, it blocks legitimate users, and if it fails, it can cut the network. Many teams run a new rule in alert-only mode first and switch to blocking after the false positives are understood.

Signature-based detection

A signature describes something known to be bad: a byte pattern in an exploit, a known malicious domain, a user-agent string used by a scanning tool, or a specific sequence of events. Signature detection is fast and gives clear explanations ("rule 2001 matched, this is a known scanner"). Its weaknesses are just as clear: it cannot catch what nobody has written a rule for, and attackers change small details to evade exact matches.

Python A tiny signature matcher

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the last request. It is a path traversal attempt written with URL encoding (%2e is a dot), and the simple signature missed it. Real systems normalize and decode input before matching, which is the same lesson as Input Validation and Sanitization: inspect the canonical form, not the raw bytes.

Anomaly-based detection

Anomaly detection works the other way around. Instead of describing bad behavior, it learns what normal looks like and flags large deviations: a server that suddenly sends ten times its usual data, a workstation talking to thousands of addresses, a login at 3 a.m. from a new country. It can catch novel attacks, but it needs a trustworthy baseline and produces more false alarms, because unusual is not the same as malicious (a big software update is also unusual).

Python A baseline and a z-score

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The z-score says how many standard deviations a value is from the baseline average. A threshold of 3 is a common starting point, and tuning it is a trade-off: lower it and you catch more but drown analysts in noise; raise it and you miss quieter attacks.

In practice most products combine both ideas: signatures for the known and cheap, anomalies for the unknown.

Approach Strength Weakness
Signature Precise, explainable, low false positives Blind to new attacks, easy to evade by small changes
Anomaly Can catch novel behavior Needs a baseline, more false positives, harder to explain

What flow logs contain

Storing every packet is expensive and often privacy-sensitive. Flow records (NetFlow, IPFIX, cloud VPC flow logs) keep only a summary of each conversation: source and destination address, ports, protocol, packet and byte counts, start time and duration, and whether the connection was accepted or rejected. They carry no payload, so they are small, long-lived and still very useful for finding scanning, data exfiltration and lateral movement.

A single flow line looks like this:

TEXT
2026-03-02T09:14:07Z 203.0.113.50 -> 10.0.0.8:22 tcp packets=3 bytes=180 action=REJECT

From records like these you can ask questions such as "which source touched the most distinct destination ports?" or "which internal host sent the most data to the outside?".

Detecting a port scan

A port scan is an attacker (or an admin) checking which ports on a host accept connections. Legitimate clients connect to one or two ports. A scanner connects to many different ports in a short time, and most attempts fail. That gives a clear detection rule: one source, many distinct destination ports, short window.

The demo below generates flow records with a fixed seed: mostly ordinary traffic, one scanner and one noisy but harmless backup server.

Python Port-scan detector on generated flow logs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The detector counts distinct destination ports, so the backup server, which touches 40 hosts but only one port, is not flagged. Choosing the right feature is what keeps false positives down. Real scanners are sneakier: a slow scan spreads probes over hours to stay under any short window, and a distributed scan uses many source addresses. Defenders answer with longer windows, grouping by target, and correlating alerts across sources in a SIEM (see Logging, Log Analysis and SIEM).

Operating a monitoring program

  • Tune rules. An untuned IDS produces thousands of alerts a day, and analysts learn to ignore them. Disable noisy rules, add allow-lists for known scanners (such as your own vulnerability scanner) and record why.
  • Place sensors where traffic matters: at the internet edge, between network segments (see Firewalls, Segmentation and Zero Trust) and in front of critical servers.
  • Encrypted traffic hides payloads from signatures. Flow metadata, DNS logs and host-based sensors fill the gap.
  • Keep logs safe. Ship alerts and flows to a separate system, because an intruder will try to erase local evidence.
  • Measure. Track how many alerts were real (true positives) and how long it took to notice and respond.

Evasion techniques against intrusion detection

Attackers try to hide traffic from signature-based sensors. Knowing the classes helps defenders tune detection.

  • Fragmentation and segmentation: splitting a payload over many small packets so no single packet matches a signature, unless the sensor reassembles streams first.
  • Encoding and obfuscation: representing the same request in other forms (Unicode or double URL encoding, mixed case, comments inside a string) so a naive pattern misses it.
  • Padding and appending: adding harmless characters to a string so a fixed-length or exact-match rule fails.
  • Custom packets and timing: unusual flags, overlapping segments or very slow "low and slow" activity that stays under thresholds.
  • Encryption: hiding content in TLS so the sensor sees only metadata.

Defenses: normalize and reassemble traffic before matching, decode inputs the way the destination would, combine signatures with behavior and anomaly rules, inspect at the endpoint where traffic is decrypted, and keep rules current. No single sensor sees everything.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main difference between an IDS and an IPS?

    The detection logic can be the same. An IPS is placed in the traffic path and can drop packets, while an IDS usually watches a copy and raises alerts.
  2. 2. Which weakness is typical of signature-based detection?

    Signatures describe known bad patterns, so new attacks or lightly modified ones can slip by. Needing a baseline and flagging spikes are characteristics of anomaly detection.
  3. 3. Why does the port-scan demo count distinct destination ports instead of the number of connections?

    The number of distinct ports separates scanning behavior from normal heavy use, such as a backup job that always connects to one port.
  4. 4. Which data do flow logs normally NOT contain?

    Flow records are summaries of conversations. They omit payloads, which keeps them small and less privacy-sensitive.

Summary

  • An IDS detects and alerts; an IPS sits inline and can also block. Network sensors see many hosts, host sensors see local activity.
  • Signature detection is precise for known threats but blind to new ones; anomaly detection finds the unusual but needs a baseline and creates false positives.
  • Flow logs summarize who talked to whom without payloads, which is enough to find scans, exfiltration and lateral movement.
  • A port-scan rule looks for one source touching many distinct ports quickly; choosing the right feature keeps false positives low.
  • Tune rules, place sensors deliberately, protect the logs and measure how well detection works.

Next: Secure Coding Principles, where the focus moves from defending networks to writing code that is hard to attack.