Skip to content
Tutorials Center
My learning

Topics

Lesson 53 of 80 ยท Malware and Endpoint Security

Detecting Malicious Behavior

Learn how defenders detect intrusions: indicators of compromise, hash and YARA-style matching, entropy checks and behavior baselines, with Python demos on synthetic data.

Prevention fails sometimes. A user clicks, a patch is late, a stolen password works. That is why defenders also detect: they look for evidence that something bad is happening and respond before it becomes a breach. Detection is a different skill from prevention. You can't list every bad thing in advance, so you combine several approaches: matching known-bad evidence, recognizing suspicious patterns, and noticing when behavior drifts from normal.

In this lesson you'll learn the vocabulary (indicators of compromise, signatures, behavior), how hash and YARA-style matching works, why entropy can reveal packed or random-looking data, and how a baseline turns "weird" into a number. All data is synthetic, and all code only reads strings you provide. It continues from Endpoint Protection and System Hardening and leads into log analysis in Logging, Log Analysis and SIEM.

Three families of detection

flowchart LR
    E["Events and files<br/>(logs, processes, network, binaries)"] --> A["Known-bad matching<br/>(IOCs, signatures)"]
    E --> B["Pattern rules<br/>(sequences, thresholds)"]
    E --> C["Anomaly detection<br/>(deviation from a baseline)"]
    A --> X["Alert"]
    B --> X
    C --> X
    X --> Y["Analyst triage"]

Each family catches different attacks, so strong programs run all three and tune them over time.

  1. Known-bad matching compares what you see with a list of things already identified as malicious. It is precise and fast, but it only finds what someone has seen before.
  2. Pattern (behavioral) rules describe suspicious activity in general terms: "a document application started a command shell", "ten failed logins then a success". They catch new variants that behave like old ones.
  3. Anomaly detection learns what is normal and flags what is not. It can find unknown attacks, but it also flags harmless oddities, so it needs careful tuning.

Indicators of compromise

An indicator of compromise (IOC) is a piece of evidence that suggests a system was compromised. Typical IOCs are:

  • file hashes of known malware;
  • IP addresses and domain names used for command and control;
  • URLs, e-mail sender addresses, subject lines;
  • file names or paths, registry keys, mutex names;
  • unusual user-agent strings or certificate fingerprints.

IOCs are shared in threat-intelligence feeds so one victim's discovery can protect others. They differ in value over time. A file hash is exact but trivial for an attacker to change: recompile, and it is a new hash. An IP address is cheap to replace. The attacker's behavior and tooling is the hardest thing to change. That ordering is sometimes called the "pyramid of pain": the higher up you detect (behaviors, tactics), the more it costs the attacker to evade you. Compare it with the techniques catalogued in The Attack Lifecycle and MITRE ATT&CK.

IOCs also expire. An address that hosted malware last month may belong to an innocent customer today. Good feeds attach dates and confidence, and good detection rules age out old entries.

Matching events against IOCs

Matching means a lookup against a set. Here is the idea on a handful of invented events. Hashes are shown shortened, and the IP addresses come from documentation ranges that are never used on the real internet.

Python Match events against an IOC list

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the .lower() call: the domain was written with capital letters in the event. Normalizing before comparing is a small habit that prevents misses. Notice also what is not caught: a brand-new file with a never-seen hash on ws-01 would sail through.

Signatures and YARA-style rules

A signature describes what a malicious file or message looks like. Antivirus engines ship millions. YARA is a widely used rule language for the same job: a rule names a set of strings or byte patterns and a condition that says how many must match. Researchers use YARA to classify malware families and to hunt across file collections.

A rule is human-readable. This one is only displayed, not run (YARA is not available in the sandbox), and it describes a made-up family:

TEXT
rule Suspicious_Script_Dropper
{
    meta:
        description = "Script that decodes a payload and runs it"
        author = "demo"
    strings:
        $a = "FromBase64String"
        $b = "Invoke-Expression"
        $c = "DownloadString"
    condition:
        2 of them
}

The idea behind it is easy to reproduce in Python: look for strings and apply a condition. Real rules also support byte patterns, wildcards, regular expressions and file-size or header checks, and they run fast across huge file sets. The next demo implements a miniature matcher over three pieces of text and shows how a threshold reduces false positives.

Python A tiny YARA-style matcher

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

backup_script.ps1 uses one suspicious command, a perfectly normal administrative thing to do, so the need: 2 threshold keeps it from being flagged. Threshold design is the heart of rule writing: too loose and analysts drown in false positives, too strict and real malware slips by. Pattern matching also has a known weakness. Attackers obfuscate: they encode, split or encrypt strings so the text never appears. That is why behavior and entropy checks matter.

Entropy: spotting data that looks random

Shannon entropy measures how unpredictable a sequence of characters or bytes is, in bits per symbol. English text has low entropy because some letters are far more common than others and some pairs repeat. Compressed, encrypted or randomly generated data looks like noise and has high entropy. Defenders use this in several places:

  • Packed or encrypted executables and embedded blobs have high entropy sections compared with normal code.
  • Algorithmically generated domains (DGAs) used by some malware look random (x7kq9zpl3vt.example), while human-chosen names (mail.example) look like words.
  • Mass file encryption: a folder whose files suddenly all become high-entropy is a strong ransomware signal.

The formula is H = - sum(p * log2(p)) over each symbol's frequency p. The code is short:

Python Entropy of text, domains and bytes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look closely at the first block: the random-looking domain scores lower than the English sentence. A 14-character string can never exceed log2(14), about 3.81 bits, so entropy on short strings says little, and real domain-generation detectors use letter-pair statistics or trained models instead. The byte blocks show the useful case, thousands of bytes, where plain text scores 3.74 and random-like data 7.91.

Treat entropy as a clue, not a verdict. Compressed formats such as ZIP, JPEG and MP4 are legitimately high entropy, and very short strings give noisy results. Real detectors combine entropy with other signals, such as file type, how many files changed in a short time, and renamed extensions.

Behavior baselines

Anomaly detection starts with a baseline: a description of normal. For each user, host or service you record a measurement (logins per hour, bytes sent per day, number of distinct processes started) and its usual range. A simple tool for "how unusual is today's value" is the z-score: the number of standard deviations from the average. Values with a high absolute z-score deserve a look.

Python Flag unusual outbound traffic against a baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A host sending several times its usual volume could be a backup job, or data being stolen. The alert does not say which; it says "look here". Good baselines are per entity (a file server and a laptop are not alike), account for weekly and seasonal rhythms, and need a quiet learning period free of attacks. Attackers who go slowly can shift a baseline, so combine anomaly alerts with known-bad matching and behavior rules. More examples of attacks that show up in event data follow in Spotting Attacks in Logs.

Good detection engineering

  • Tie rules to behaviors, such as ATT&CK techniques, not just single indicators.
  • Measure quality. Track false positives and what each rule found. Retire noisy rules, and tune thresholds with data.
  • Test detections. Replay safe, simulated activity in a lab and confirm the rule fires.
  • Write down the response. An alert without a documented next step wastes the analyst's time. See The Incident Response Process.
  • Keep telemetry. You can only detect what you log; see the next lesson on Logging, Log Analysis and SIEM.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why are file hashes a weak long-term indicator of compromise?

    Hashes are exact, so any rebuild or repack creates a new one. Indicators about behavior and tooling are harder for attackers to change.
  2. 2. A YARA-style rule says 'need 2 of 3 strings'. What is the purpose of the threshold?

    Requiring more than one match reduces false positives, because legitimate scripts often contain a single suspicious-looking command.
  3. 3. Which situation is most likely to produce very high entropy?

    Encrypted data looks like random noise, so each byte value is about equally likely. That gives entropy close to 8 bits per byte.
  4. 4. What is a major drawback of anomaly-based detection?

    Anomaly detection finds deviation, not malice. Backups, a new project or a holiday can all look unusual, so analysts must triage.
  5. 5. What does a z-score of 4.5 for today's data volume mean?

    The z-score is the distance from the mean measured in standard deviations. A large absolute value means the observation is far outside the host's normal range.

Summary

  • Detection complements prevention and combines known-bad matching, behavioral pattern rules and anomaly detection.
  • IOCs (hashes, IPs, domains, file names) are quick to match but easy for attackers to change and expire quickly; behavior is harder to evade.
  • Signature and YARA-style rules combine strings with a condition, and thresholds balance false positives against misses.
  • Entropy highlights random-looking data such as packed code, generated domains and mass encryption, but is only a clue.
  • Baselines and z-scores turn "unusual" into a measurable alert, and every detection needs testing, tuning and a documented response.

Next: Logging, Log Analysis and SIEM, where you'll gather the data that all of these detections depend on.