Skip to content
Tutorials Center
My learning

Topics

Lesson 54 of 80 ยท Malware and Endpoint Security

Reverse Engineering Basics

Learn what reverse engineering is for in security, the levels of abstraction, static versus dynamic analysis, common tools and safe lab habits, with Python demos.

Reverse engineering (RE) means taking a finished product and working out how it was built and what it does, without access to the original design or source code. In security it is how analysts answer questions such as: What does this suspicious file do? How does it talk to its controller? What exactly does this patch fix? Is this closed-source library safe to ship?

You do not need to be an assembly expert to understand the ideas. This lesson explains why defenders reverse engineer, the levels at which a program can be viewed, the difference between static and dynamic analysis, the tools analysts use, and the safety habits that keep a lab from becoming an incident. The hands-on parts use Python, because Python code can be inspected at the same levels as compiled programs and runs safely in the browser.

Only analyze software you are allowed to analyze: your own code, samples from a lab you control, or material permitted by license and law. Never run unknown samples on a machine that holds real data or has network access.

Why defenders reverse engineer

  • Malware analysis: learn what a sample does, extract indicators of compromise (IOCs) such as domains, file names and registry keys, and write detections. See Detecting Malicious Behavior.
  • Vulnerability research: find and fix flaws in your own or third-party software, or understand a vendor patch by comparing the old and new binaries ("patch diffing").
  • Incident response and forensics: explain how an intruder's tool worked (Digital Forensics Basics).
  • Interoperability and legacy: understand an undocumented file format or protocol, or recover logic from software whose source is lost.
  • Software protection: to design good obfuscation or licence checks you must know how attackers will analyze them.

Levels of abstraction

A program exists at several levels, and reverse engineering is moving up from the lowest one the analyst can reach.

Level What you see Example
Source code Names, comments, structure if user == "admin":
Intermediate or bytecode Instructions for a virtual machine, some names kept Python bytecode, Java .class files, .NET IL
Assembly CPU instructions, registers, jumps cmp eax, 5 / jne 0x401020
Machine code Raw bytes 83 F8 05 75 0E

A disassembler turns machine code into assembly; a decompiler goes further and guesses at source-like code. Decompiled output is an approximation: variable names and comments are gone, and the compiler's optimizations rearrange logic. The higher the level the file keeps, the easier the analysis. That is why bytecode languages are much easier to analyze than natively compiled programs, and why malware authors pack or obfuscate their code.

Static analysis

Static analysis examines a program without running it. It is safe and repeatable, and it covers all code paths, including ones that never execute in your test. Typical steps:

  1. Identify the file. Check the real type (header bytes, not the extension), size and a cryptographic hash. The hash is how you search threat-intelligence databases (Hash Functions and Integrity).
  2. Extract strings. Readable text such as URLs, file paths, error messages and command names reveals intent.
  3. Inspect structure: imported functions (does a calculator import network functions?), embedded resources, sections with high entropy (a sign of packed or encrypted data).
  4. Read the disassembly or decompiled code to follow the logic.
Python Static triage: hash and strings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Nothing was executed, yet three behaviours are already suspected: it contacts a URL, drops a file and sets itself to start at login. An analyst records these as hypotheses to confirm. Each string is only a clue: malware can hide its strings by encrypting them, which is itself a finding.

Dynamic analysis

Dynamic analysis runs the program in a controlled environment and watches what it does: files created, registry or configuration changes, processes started, network connections, and API calls. It reveals behaviour that is hard to see in code, such as strings decrypted at run time, but it only shows the paths that actually executed, and some malware detects sandboxes and behaves innocently.

Tools used here are debuggers (pause execution, step through instructions, inspect memory), system monitors (file, registry and process events), network monitors, and sandboxes that automate the run and produce a report.

You can try the idea in miniature with Python's tracing hooks. This function stands in for an unknown routine; the tracer logs every call it makes without us reading its source:

Python Dynamic analysis: watch calls at run time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The trace shows that the routine reverses its input and then upper-cases it, without us reading the code. A real debugger or sandbox does the same for system calls and network activity. The best analyses combine both: static analysis tells you what could happen, dynamic analysis confirms what did.

Seeing the same function at two levels

Python compiles source to bytecode that you can disassemble with the standard dis module. This is the same relationship as source versus assembly, just friendlier.

Python Source versus bytecode

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Even without the source, the constants list exposes the hard-coded PIN. This is the lesson for developers: anything shipped to a user's device can be read. Secrets, licence rules and "hidden" API keys in a client application or mobile app are not secret (see Secrets, Logging and Error Handling and Mobile Application Security). Compare against a hash on a server instead.

Common tools

Purpose Examples
Disassembly and decompilation Ghidra, IDA, Binary Ninja, radare2/Cutter
Debugging x64dbg, WinDbg, GDB
File and string inspection file, strings, hex editors, PE viewers
Behaviour monitoring Process and file monitors, packet capture (Network Traffic Analysis)
Bytecode and managed code dis for Python, Java decompilers, .NET decompilers
Sandboxes Automated analysis environments run by vendors or in your own lab

Building a safe analysis lab

  • Use a virtual machine with snapshots so you can revert after every run.
  • Isolate the network: host-only or a simulated internet, never your real LAN.
  • Keep no personal data, credentials or shared folders on the analysis machine.
  • Work on copies, record hashes, and write notes as you go so findings can be reproduced.
  • Handle samples with care: rename or archive them with a password so nothing launches by accident.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main advantage of static analysis?

    Static analysis never executes the sample, so it is safe and can reach every branch. Its weakness is that obfuscation and packing can hide the real logic.
  2. 2. Why is bytecode (such as Java or Python) generally easier to reverse engineer than natively compiled code?

    Bytecode formats retain class, function and often variable names and a clean instruction set, so decompilers produce readable results.
  3. 3. A sample behaves harmlessly in your sandbox but harmfully elsewhere. What explains this best?

    Some malware checks for virtual machines, debuggers or missing user activity and stays quiet. Combining dynamic with static analysis helps expose this.
  4. 4. What should you do before running an unknown sample for dynamic analysis?

    An isolated, revertible environment protects your real systems and lets you repeat the analysis.

Summary

  • Reverse engineering recovers how software works without its source; defenders use it for malware analysis, vulnerability research, incident response and interoperability.
  • Programs exist as source, bytecode, assembly and machine code; analysts move upward using disassemblers and decompilers.
  • Static analysis (hashes, strings, imports, disassembly) is safe and broad; dynamic analysis (debuggers, monitors, sandboxes) shows real behaviour. Use both.
  • Packing, encryption and sandbox detection are the main obstacles.
  • Anything shipped to a client can be read, so never hide secrets in distributed code.
  • Work in an isolated lab, on copies, with snapshots and notes.

Next: Detecting Malicious Behavior, to turn what you learn from analysis into detection rules.