Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 53 ยท Systems and Post-Exploitation Concepts

Reverse Engineering Basics

Learn what reverse engineering is, how static and dynamic approaches differ, how disassembly and decompilation work, which tools are used and how the work stays legal and ethical.

When you have a program but not its source code, reverse engineering (RE) is the practice of working out what it does and how. Security people reverse programs to analyze malware, to find vulnerabilities in software they are allowed to examine, to check that a patch really fixes a flaw, to understand an undocumented file format or protocol, and to recover how a legacy system works when documentation is gone. It is slow, detailed work, but its core ideas are accessible.

Reverse engineering can conflict with software licenses and, in some jurisdictions, with laws that protect copy-protection or restrict decompilation. Work only on software you own, software you have explicit permission to analyze, malware samples you are authorized to handle (in an isolated lab, see Dynamic Analysis and Malware Lab Setup), or open crackme-style practice challenges built for learning. Never use RE to pirate software or to defeat licensing. See Law, Ethics and Authorization.

From source code to machine code

A compiler translates human-readable source into machine code, the instructions a processor executes. Along the way, names of local variables, comments and structure are mostly lost. What remains is a sequence of low-level operations. Reverse engineering reconstructs meaning from those operations.

flowchart LR
  A["Source code<br/>(readable)"] -->|compile| B["Machine code<br/>(binary)"]
  B -->|disassemble| C["Assembly<br/>(instruction mnemonics)"]
  C -->|decompile| D["Pseudo-code<br/>(approximate source)"]

Disassembly is exact but low-level. Decompilation is easier to read but approximate.

  • A disassembler turns bytes back into assembly instructions such as mov, add, cmp, jmp, call. Each line corresponds to exactly what the processor will do.
  • A decompiler goes further and guesses C-like code from patterns. It is quicker to read but can be wrong, and it loses names unless symbols are present.
  • Managed languages (Java, C#, some Python packaging) keep a lot of structure in bytecode, so decompiling them is often much more accurate than decompiling native code.

Static versus dynamic reverse engineering

  • Static RE studies the program without running it: reading disassembly, following references, looking at strings and imports. It is safe and complete in principle but slow, and obfuscation can hide logic (see Static Malware Analysis).
  • Dynamic RE runs the program under a debugger and watches it. You set breakpoints to pause at an address, step through instructions, inspect registers and memory and see real values, for instance the decrypted text that static reading could not reveal. It is powerful but risky for hostile code, so it belongs in a lab.

Most real work mixes the two: static reading to understand the structure, dynamic runs to confirm what a function does with real input.

What you look at

Even without knowing all of assembly, a few ideas carry you far.

  • Registers and the stack. The processor has tiny fast storage called registers; function arguments and local variables live in registers and on the stack. Calling conventions define which is which.
  • Control flow. cmp plus a conditional jump is an if. A jump back to an earlier address is a loop. A graph view of basic blocks (straight-line code) and edges makes this visible.
  • Function calls. A call to a library function (for example one that opens a network connection or a file) tells you more than ten lines of arithmetic.
  • Strings and constants. A reference to an error message such as "Wrong password" in a validation routine points straight to the check you care about.
  • Data structures. Repeated access at fixed offsets from a pointer suggests a struct.

Tools

  • Disassemblers and decompilers: Ghidra (free, from the NSA's research group, open source), IDA (commercial), Binary Ninja, radare2 and Cutter.
  • Debuggers: x64dbg and WinDbg on Windows, GDB (often with helper plugins) on Linux, LLDB.
  • Utilities: strings, objdump, readelf, ltrace and strace on Linux; hex editors; file format parsers.
  • Managed code: dnSpy-style .NET decompilers and Java decompilers.

On Kali several of these are packaged, but Ghidra runs happily on ordinary machines too.

A tiny disassembler stand-in

Real disassembly depends on the processor's encoding. The stand-in below defines an invented toy instruction set with a handful of opcodes, then writes a disassembler and a stepping "debugger" for it, so you can see exactly what the real tools do conceptually: decode bytes into mnemonics and execute them while showing state. The program checks a secret number, a typical crackme-style check.

Node.js Disassemble and step a toy program

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The static step (reading LOAD x, ADD 3, CMP 10) lets you work out the answer without running anything; the dynamic trace confirms it. That is the whole pattern of a crackme in miniature, and it is also how analysts find the check in a license routine they own.

Obfuscation and anti-analysis

Authors of malware and some commercial software try to slow analysis: packing and encryption, control-flow flattening (rewriting ifs into a confusing dispatch loop), junk instructions, anti-debugging checks (detect whether a debugger is attached) and virtual machine detection. Analysts respond by unpacking, tracing execution and patching the checks in a lab. Knowing this shows why RE skills reward patience.

A strings and constants hunt

A faster first look is looking for interesting clues without reading every instruction. The second stand-in scans the toy program's messages and constants, the sort of cross-reference a real tool offers ("show me where this string is used").

Node.js Cross-reference strings and constants

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Using RE for defense

Reverse engineering is not only offensive. Defenders use it to understand a malware sample's indicators, to diff a vendor patch against the vulnerable version and learn what was fixed, to verify that a binary in the supply chain matches what its vendor claims, and to teach developers how little their compiled code hides. The last point matters: secrets in client binaries (API keys, passwords, license logic) are not secret. Keep them on the server. See Secure Development.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the difference between a disassembler and a decompiler?

    Disassembly is an exact view of the instructions. Decompilation reconstructs approximate source and may be wrong.
  2. 2. What does dynamic reverse engineering use that static does not?

    Running the code under a debugger shows real values, such as decrypted strings, that are hard to see statically.
  3. 3. Why should sensitive keys not be embedded in a client application?

    Compiled code is analyzable. Secrets must stay on a server that you control.
  4. 4. Which is an appropriate target for practicing reverse engineering?

    Practice only on software you own, are permitted to analyze or that is designed for learning.

Summary

  • Reverse engineering reconstructs how a program works without its source, for malware analysis, vulnerability research, patch analysis and compatibility work.
  • Disassemblers show exact instructions; decompilers produce approximate high-level code.
  • Static RE reads the code; dynamic RE uses a debugger to watch it run. Real work mixes both.
  • Control flow, calls, strings and constants are the first clues; tools include Ghidra, IDA, Binary Ninja, radare2 and debuggers such as x64dbg and GDB.
  • Obfuscation, packing and anti-debugging slow analysis but do not stop it.
  • Respect licenses and law, work in isolated labs for hostile code and never rely on secrets in client binaries.

Next: Persistence, Lateral Movement and Detection