Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 37 ยท Behavioural Indicators and YARA

Behavioural Patterns and API Call Sequences

Detect malware by what it does: map API call sequences to capabilities such as persistence, injection and exfiltration, and score made-up traces with Python stand-ins.

Hashes and strings describe what a sample is. Behaviour describes what it does, and that is far harder for an author to change. A new build can alter every byte, yet it still has to start itself after a reboot, talk to a server and read files. Detections built on behaviour sit at the top of the pyramid of pain from the indicators lesson.

The usual raw material is a trace: an ordered list of API calls, file operations, registry changes and network events recorded during dynamic analysis. This lesson shows how to turn a trace into capabilities. Everything is read-only reasoning over invented traces; the Python programs are stand-ins for the rule engines inside sandboxes and endpoint detection products, and they never touch a real system.

From calls to capabilities

Trace Ordered events from a run.
Pattern A sequence that means something.
Capability Persistence, collection, C2.

One call is noise. A sequence is a signal.

1 / 2

Single calls are weak signals

Calls such as CreateFileW, RegSetValueExW or InternetOpenA appear in countless legitimate programs, so flagging them alone buries you in false positives. What matters is the combination, order and arguments:

  • Writing to the registry's Run key is normal for an installer, suspicious for a document viewer.
  • Allocating executable memory is normal for a browser's JIT, suspicious in a small file that has no reason for it.
  • Making an HTTP request is normal, but doing so every 60 seconds to a never-before-seen domain with a fixed user agent is a beacon.

Common capability patterns

Analysts keep a catalogue of patterns. Here are the main groups, with the kind of calls and artefacts that point to each. This is a defensive catalogue to recognise behaviour; it is not a recipe.

Capability What a trace may show
Persistence Writes to Run/RunOnce keys, new scheduled tasks or services, files dropped into the Startup folder, WMI event subscriptions
Process injection A process opening another process, allocating memory in it, writing into it and starting a thread there (typically OpenProcess -> VirtualAllocEx -> WriteProcessMemory -> CreateRemoteThread or similar)
Defence evasion Checks for debuggers or virtual machines, deletion of its own file, disabling security tools, clearing logs
Discovery Enumerating processes, users, network adapters, drives, installed software
Collection Reading browser profile folders, document files, keystroke or clipboard APIs
Command and control Periodic network traffic, DNS lookups to algorithmically odd names, custom protocols or headers
Ransom-like behaviour Rapid reading, rewriting and renaming of many files with a new extension, deleting backup copies, creating a note file in many folders
Exfiltration Large outbound transfers after a collection phase, archive creation before uploads

The persistence and injection rows connect directly with the packing clues: unpacked payloads often inject or persist as their first act.

Matching an ordered sequence

A simple rule says: "these events must occur in this order, not necessarily next to each other." Between steps a sample may run harmless calls, so rules test for an ordered subsequence. The stand-in below defines three rules and checks them against a made-up trace.

Python Match ordered API patterns in a trace

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first three rules match, even though Sleep sits inside the injection pattern; the fourth does not, because the trace has no MoveFileW. The step in it trick works because testing membership on an iterator consumes it up to the match, which enforces the order.

Arguments matter

A pattern rule becomes much more precise when it also tests arguments. "Writes a registry value" is common. "Writes a value under ...\CurrentVersion\Run whose data points into a user-writable folder" is far more telling. Likewise, an outbound connection matters more when the destination is a raw IP, a freshly registered domain or an unusual port.

Rules in tools such as sandboxes and Sigma (a generic format for log-based detections) test field values with simple comparisons. The next stand-in scores events from a made-up trace with weights, a style often used in sandbox "verdict" engines.

Python Score events with argument checks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two Run-key writes get different scores: the application-settings one scores nothing, while the autorun entry that launches an executable from a public folder scores five. The direct-IP connection on an unusual port adds three more. This "weighted evidence" approach is not a replacement for analyst judgement, but it makes verdicts explainable: every point has a reason.

Behaviour in logs, not just sandboxes

Once you know the pattern, you can look for it in production telemetry: Windows event logs (process creation with command lines, service installation, scheduled task creation), Sysmon events, EDR (endpoint detection and response) telemetry and proxy or DNS logs. A rule written in the generic Sigma format can be converted to the query language of most SIEM products:

TEXT
title: Executable launched from a public user folder by a script host
logsource:
    category: process_creation
    product: windows
detection:
    selection:
        ParentImage|endswith: '\wscript.exe'
        Image|contains: '\Users\Public\'
    condition: selection
level: medium

This is a read-only example. The next lesson, mapping to ATT&CK, connects such behaviours to a shared vocabulary so detections and reports speak the same language.

Write down for each rule what legitimate software could trigger it. If you cannot name an example, you have not tested enough; if you can, add an exclusion or a stricter argument check.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why are single API calls weak indicators?

    APIs like CreateFileW or InternetOpenA are everywhere; sequences and arguments make a signal.
  2. 2. What does an ordered-subsequence rule require?

    Other harmless calls can sit between steps, but the order of the pattern must hold.
  3. 3. Which trace most clearly suggests process injection?

    Opening another process, allocating and writing memory in it and starting a thread is the classic injection sequence.
  4. 4. What is Sigma?

    Sigma describes detections once and tools convert them to the query language of each platform.

Summary

  • Behaviour is harder for authors to change than bytes, so behaviour-based detection lasts longer.
  • Sequences, order and arguments turn noisy single events into meaningful patterns.
  • Catalogue patterns by capability: persistence, injection, evasion, discovery, collection, C2, ransom-like and exfiltration.
  • Ordered-subsequence matching and weighted scoring are the core of many sandbox verdicts.
  • The same patterns apply to production logs through Sysmon, EDR and Sigma rules.
  • Always document what legitimate software could trigger a rule.

Next: Writing Up Your Findings