Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 37 Β· Exploits and Payloads

Encoders, Evasion Testing and the Defender's View

Understand what encoders and evasion modules do, why encoding is not real protection, how signature and behaviour detection work, and how to test your own defences.

When people first learn that Metasploit has encoders and evasion modules, they often think of them as invisibility tools. They are not. Encoders exist for a mundane technical reason, and modern defences detect the output of both easily. For a defender, the useful question is the opposite: how do I check that my defences catch what attackers commonly try?

This lesson explains the concepts at a high level and shows how detection works, using harmless stand-ins. It does not describe how to defeat any specific product.

Why encoders exist

Exploits often cannot carry certain bytes. A string-handling bug might stop at a zero byte; a web request might forbid some characters. An encoder rewrites a payload so it contains only allowed bytes and adds a tiny decoder that restores the original at run time. That was the main job of encoders: compatibility.

The side effect is that the bytes look different. Early antivirus matched exact byte patterns, so encoding changed what the scanner saw. Defenders responded quickly:

  • Scanners emulate the decoder and examine the decoded result.
  • Decoder stubs themselves became signatures.
  • Behaviour monitoring looks at what a program does, not what it looks like.

The result: a payload encoded many times is often more suspicious, not less. Treat encoders as a compatibility feature.

flowchart LR
    P["Payload bytes"] --> E["Encoder"]
    E --> S["Encoded bytes + decoder stub"]
    S --> D{"Defence"}
    D -- "signature" --> X["Matches known stub or decoded content"]
    D -- "emulation" --> Y["Decodes in a sandbox and inspects"]
    D -- "behaviour" --> Z["Watches what runs"]

Hiding bytes does not hide behaviour.

What evasion modules are for

Evasion modules generate files that try to slip past common endpoint protections. In a legitimate engagement they answer one question: would the client's defence have stopped this class of test file? If it did not, that is a finding with a concrete fix: update signatures, enable behaviour protection, restrict script engines, apply application allow-listing.

Treat the results carefully: a bypass means "your defence has a gap today", and the gap changes when vendors update. Record the product, version and date.

Three layers of detection

How defences catch it

Signatures Known bytes and strings.
Emulation Run a copy in a sandbox.
Behaviour Watch actions at run time.
Network Spot callbacks and beacons.

Good defence stacks all four.

1 / 2

A toy signature scanner

Signature detection is simple at its core: look for known byte patterns. The stand-in below does that with harmless text instead of code. It then shows the classic weakness (a trivial change defeats a naive signature) and the classic answer (normalise before matching). It contains no malicious content: the "pattern" is just a marker string.

Python Why naive signatures are fragile

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real products go much further, but the principle holds: once a scanner decodes common encodings, encoding stops helping.

Behaviour beats appearance

Behaviour rules look at sequences of actions. Here is a tiny rule engine over made-up process events: "a document viewer starts a command interpreter" is suspicious regardless of any file's bytes.

Python A behaviour rule over process events

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The rule alerts on two events whatever the payload looked like. That is why testers' evasion results matter less than whether behaviour rules fire.

A practical purple-team test plan

  1. Pick one technique (for example, "a service process starts a shell").
  2. Run it in the lab against a host with the production endpoint agent installed.
  3. Record: Did the agent block it? Did it alert? How long until the alert reached a human?
  4. If not blocked, add or tune a rule; run the test again.
  5. Keep the test as a regression check: run it after each major change.

Defence checklist

  • Keep endpoint protection and signatures updated, with behaviour protection and exploit mitigations enabled.
  • Use application allow-listing for servers; unknown binaries should not run.
  • Apply least privilege to service accounts and block interpreters where they are not needed.
  • Log process creation and forward logs; test that alerts reach someone.
  • Control egress and inspect outbound traffic metadata.
  • Retest regularly: detection quality changes with every product update.

Check your understanding

Quick check Β· 4 questions

of 4 answered You scored / 4 Β· perfect!

  1. 1. What was the original main purpose of encoders?

    Encoders remove forbidden bytes; the changed look was a side effect.
  2. 2. Why does encoding a payload many times often fail?

    Emulation and signatures on stubs defeat most encoding.
  3. 3. What does an evasion module help a legitimate tester learn?

    It measures detection so gaps can be closed.
  4. 4. Which detection layer ignores how a file looks and watches what happens?

    Behaviour monitoring watches actions such as unusual parent-child processes.

Summary

  • Encoders solve byte-compatibility; they are not a reliable disguise.
  • Evasion modules are for measuring defences and finding gaps.
  • Defence is layered: signatures, emulation, behaviour and network monitoring.
  • Turn each test into a repeatable detection check and fix what is missed.

Next: Meterpreter Overview: Sessions and Staging