Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Projects and Practice

Project: A Byte-Entropy Analyzer

Build a tool that slides a window over a file, computes Shannon entropy per byte, and classifies regions as text, zeros, compressed or random, a trick used in forensics and malware analysis.

In this project you build a small entropy analyzer, a tool that reads a stream of bytes and tells you what kind of data each region probably contains. Security analysts use the same idea to find encrypted or packed sections inside executables, forensic tools use it to spot compressed archives inside disk images, and storage software uses it to decide whether compression is worth trying.

The principle comes straight from the course: Shannon entropy of the byte distribution, H = -sum p(b) log2 p(b), is between 0 bits per byte (constant data) and 8 bits per byte (uniformly random bytes). Different kinds of data fall into characteristic ranges, and, as you learned in DEFLATE, compressed or encrypted data looks random.

Step 1: entropy of a byte string

We use NumPy's bincount to count the 256 possible byte values quickly.

Python Step 1: the entropy function

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Base64 uses 64 symbols, so its entropy per byte is at most log2(64) = 6 bits even when the payload is random. That is a useful signature: a plateau near 6 bits suggests an encoded binary blob.

Step 2: a file with different regions

We build a test "file" from five regions of known kind, so we can check the analyzer against the truth.

Python Step 2: build a mixed test file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 3: sliding-window analysis and classification

Now the analyzer: slide a window of 1024 bytes in steps of 1024, compute its entropy, and label it by thresholds. The thresholds below come from the values in step 1 and are a reasonable starting point, not a law of nature.

Python Step 3: the analyzer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The classifier separates constant, text-like, encoded and random data. But look at the compressed (zlib) and random windows: both give values near 7.8 bits. Entropy alone cannot tell compression from encryption or from noise, because good compressors and ciphers both produce data that looks uniformly random. Real tools add other signs: file headers (magic numbers), chi-square tests of uniformity, and checks of structure such as zlib header bytes.

A 1024-byte window can hold at most 1024 different byte values, so even perfectly random data scores slightly below 8 bits (about 7.8). Bigger windows get closer to 8. This is the same finite-sample bias you met in entropy of text and DNA.

Step 4: a randomness check beyond entropy

A chi-square test compares the observed byte counts to the uniform expectation. Truly random data gives a statistic near 255 (the degrees of freedom); structured data gives huge values.

Python Step 4: chi-square uniformity score

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The random sample scores near 255, text scores in the hundreds of thousands. A zlib stream falls in between: it is close to uniform but its first bytes (header) and Huffman code structure leave a small imprint. Combining entropy with a chi-square test gives a more reliable detector than either alone.

Ideas to extend the project

  • Read a real file with open(path, "rb").read() (outside the browser) and draw an entropy map with Matplotlib.
  • Add a windowed classifier that merges neighbouring windows with the same label into regions.
  • Use the byte-pair (order-1) conditional entropy to separate compressed from encrypted data in some formats.
  • Report the "compressibility": how many bytes zlib would save for each region, to decide where compression is worthwhile.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the maximum Shannon entropy per byte?

    A byte has 256 possible values, so the maximum is log2(256) = 8 bits, reached by uniformly random bytes.
  2. 2. Why does base64-encoded random data score about 6 bits per byte?

    Each output byte is one of 64 symbols chosen nearly uniformly, giving at most 6 bits.
  3. 3. Why can't entropy alone distinguish compressed data from encrypted data?

    Good compressors and ciphers both remove statistical structure, so byte-level entropy is near the maximum for both.
  4. 4. What does a chi-square statistic near 255 on 256 byte values suggest?

    With 255 degrees of freedom, a uniform distribution gives a statistic near 255; large values signal structure.

Summary

  • Byte entropy ranges from 0 (constant) to 8 bits (random).
  • Sliding a window over a file gives an entropy map that reveals regions of different kinds.
  • Compressed, encrypted and random data all sit near 8 bits; encodings like base64 plateau lower.
  • Small windows bias entropy downward; larger windows are more accurate.
  • Combine entropy with other tests (chi-square, headers) for reliable classification.

Next: Project: An Error-Correction Simulator