Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 37 ยท Projects and Practice

Project: Build a YARA-Style Scanner

Build a small rule-based file scanner in Python with text and hex patterns, conditions and a goodware test, then measure its false positives like a real detection engineer.

In the YARA lessons you read rules and tested them. In this project you build the engine behind them: a small scanner that loads rules written as data, evaluates them on files and reports matches with offsets. Building it teaches what YARA does for you, why its design choices (fixed prefixes, early header checks, counts) exist, and how a rule's quality is measured.

The real YARA cannot run on this site, and this scanner is deliberately much simpler: it supports text strings, hex strings with ?? wildcards, a few modifiers and conditions of the form all, any or N of. The files it scans are generated in memory and harmless, so everything is safe and repeatable.

Scanner design

Rules as data Dictionaries, easy to extend.
Matchers Text, hex, case, wide.
Conditions all, any, N of, size.
Test set Bad and clean files.

Four small pieces make a scanner.

1 / 2

Step 1: the matchers

Each string in a rule has a kind. The matcher functions below return the offsets where a pattern occurs. Text strings can be case-insensitive or wide (UTF-16LE); hex patterns accept ?? as "any byte". Run the block to see the offsets for a made-up buffer.

Python Step 1: pattern matchers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 2: rules as data and a condition evaluator

A rule is a dictionary: a name, its strings and a condition. The condition is a small structure instead of a text language, which avoids writing a parser and keeps the project focused. The evaluator computes which strings match, then applies the condition. Include the header check and file size limit, as real rules do.

Python Step 2: rules, conditions and a scan function

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sample matches DemoLoader_Family, the script matches the second rule, and the clean file matches nothing. Look at the structure: the header check and size limit run before any string search, the same early-exit idea that makes real YARA fast.

Step 3: test against corpora

A scanner is only as good as its measured behaviour. Generate a set of variants of the family (some lacking strings, some changed in unrelated bytes) and a larger set of clean files with random content and occasional lookalikes, then compute true positives, false negatives, false positives, precision and recall as in rule tuning. The file generator is seeded, so the numbers repeat.

Python Step 3: measure the scanner on bad and clean files

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The any rule catches the most variants but also triggers on the clean lookalikes. 2 of 3 removes those false positives at the cost of missing a few variants that dropped two markers. Choosing between them is a judgement about what a false alarm costs compared with a miss, which is a conversation you will have with the people who run the rule. In production, blocking rules favour precision, and hunting rules favour recall.

Step 4: report matches like a tool

Real scanners print the rule name, file and matching strings with offsets, so analysts can jump straight to the evidence. Format your results in the same shape as yara -s:

Python Step 4: print matches in YARA-like output

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Challenge: add filesize conditions, a count condition (#a > 3), a fullword modifier and a rule loader that reads a simple text format. Then compare your behaviour with the real YARA on your own lab machine.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the scanner check the magic bytes and size before searching strings?

    Cheap checks first keep scanning fast across millions of files.
  2. 2. A 2 of 3 rule has fewer false positives than an any rule. What is the cost?

    Stricter conditions trade recall for precision.
  3. 3. What does the ?? wildcard in a hex pattern match?

    ?? stands for exactly one unknown byte, useful where a key or address changes.
  4. 4. Why test the scanner on clean files as well as malicious ones?

    Only clean data can reveal rules that fire on legitimate software.

Summary

  • A rule engine needs matchers, rules as data, a condition evaluator and a test corpus.
  • Cheap checks (header magic, size) run first; string searches follow.
  • Hex wildcards and wide/nocase modifiers let one rule cover variants.
  • Measure true positives, false negatives, false positives, precision and recall on seeded corpora.
  • Format output like real tools so analysts can jump to evidence.
  • This scanner is a teaching model; use the real YARA in your own lab.

Next: Project: A Beacon Detector