Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 40 ยท Tools and Toolchain

Debugging at the Instruction Level

Debug at the instruction level with gdb and lldb: disassemble, stepi, info registers, x/ memory dumps, breakpoints and crashes, plus a runnable toy debugger.

When a C program crashes, a source-level debugger shows you the line. When there is no source (a library, a stripped binary, a compiler bug, a mysterious crash inside the standard library) you need to debug the way the CPU sees the program: one instruction at a time, looking at registers and raw memory. That is instruction-level debugging, and it is the best way to turn assembly knowledge into practical skill.

This lesson covers the core commands of gdb (the GNU debugger) and lldb (the LLVM debugger used on macOS), the workflow for reading a crash, and a toy single-step debugger written in JavaScript so you can practise the same moves in your browser. The real debugger sessions are read-only; the sample output is illustrative and your addresses will differ.

Compile with symbols, then start the debugger

For learning, build a small program with debug information and no optimization, so the instructions follow the source closely:

Terminal
gcc -g -O0 -o demo demo.c
gdb ./demo            # or: lldb ./demo

-g adds symbol and line information, which lets the debugger show source lines and variable names next to the instructions. Without it, you still get addresses and instructions, only fewer labels. Debuggers work on any executable, including ones you did not compile.

The essential commands

Debuggers share most ideas but use different spellings. The table lists the ones you will use constantly.

Task gdb lldb
Run the program run (r) run (r)
Break at a function break main (b main) breakpoint set -n main (b main)
Break at an address break *0x401136 breakpoint set -a 0x401136
Continue continue (c) continue (c)
Step one instruction stepi (si) thread step-inst (si)
Step over a call (one instruction) nexti (ni) thread step-inst-over (ni)
Disassemble disassemble disassemble (di)
Show registers info registers register read
One register print $rax or p/x $rax register read rax
Examine memory x/8xb $rsp memory read --size 1 --count 8 $rsp
Backtrace backtrace (bt) bt
Show the instruction display/i $pc disassemble --pc

By default gdb shows AT&T syntax. Run set disassembly-flavor intel for Intel syntax (add it to ~/.gdbinit to keep it). Everything below uses Intel syntax.

Here is a short, illustrative gdb session on a tiny function (output trimmed, addresses will vary on your machine):

TEXT
(gdb) break add3
Breakpoint 1 at 0x1139: file demo.c, line 2.
(gdb) run
Breakpoint 1, add3 (a=1, b=2, c=3) at demo.c:2
(gdb) disassemble
Dump of assembler code for function add3:
   0x0000000000001129 <+0>:   push   rbp
   0x000000000000112a <+1>:   mov    rbp,rsp
   0x000000000000112d <+4>:   mov    DWORD PTR [rbp-0x4],edi
=> 0x0000000000001130 <+7>:   mov    DWORD PTR [rbp-0x8],esi
   ...
(gdb) stepi
(gdb) info registers rip rsp
rip            0x1133              0x1133 <add3+10>
rsp            0x7fffffffe0c0      0x7fffffffe0c0
(gdb) x/4xw $rsp
0x7fffffffe0c0: 0x00000001 0x00000002 0x00000003 0x00000000

The => marker shows the instruction the CPU is about to execute (the one at rip). Each stepi runs exactly that one instruction and stops again.

Reading memory with x

The x command (examine) is how you look at raw bytes. Its format is x/NFU address, with three optional parts:

  • N: how many units to show.
  • F: the format: x hex, d decimal, u unsigned, c character, s string, i instruction.
  • U: the unit size: b byte, h halfword (2 bytes), w word (4), g giant word (8).

Some practical examples:

TEXT
x/8xb $rsp       # 8 bytes at the stack pointer, in hex
x/2xg $rsp       # two 8-byte values (quad words)
x/s 0x402004     # a null-terminated string
x/5i $pc         # next 5 instructions
x/4dw &array     # four 32-bit integers in decimal

Remember little-endian when you read the bytes: x/4xb on the 32-bit value 0x12345678 shows 78 56 34 12, while x/xw shows the assembled value 0x12345678 again.

Breakpoints and single-stepping

A breakpoint stops the program when execution reaches an address. Debuggers implement it by temporarily replacing the first byte of the instruction with a special one-byte trap instruction, int3 (0xCC on x86), so the CPU raises an exception that the debugger catches; it then puts the original byte back. A hardware watchpoint (watch variable in gdb) uses CPU debug registers to stop whenever a memory location changes, which is very effective for finding who overwrites a value.

stepi executes one instruction. nexti does the same, but treats a call as a single step, running the whole called function. Source-level step and next do the same for lines of C. At the instruction level you are always in control of exactly what happens next, which is what makes it good for learning how calls and stack frames really work.

The next demo is a toy debugger. It runs a tiny program (summing 4+3+2+1 in a loop and storing the result in memory) and accepts commands that look like gdb's: break, run, stepi, info registers, x/Nb and disassemble. Read the command script at the bottom and predict what the output will show before running it.

JS A toy single-step debugger

Press Run (or Ctrl + Enter) to see the console output here.

No output.

After the first run, the breakpoint fires at the top of the loop with rcx=4 and rax=0. On the second continue it fires again after one pass: rax is now 4 (0 plus 4) and rcx is 3. Two stepi commands then execute add and dec, and you can watch the flag and registers change after each. The final x/4b 16 shows zeros because the store at instruction 6 has not run yet; try adding more continue and stepi lines to watch it appear. That "run a little, look, run a little" cycle is how you build understanding of any unfamiliar code.

Reading a crash

When a program crashes, the operating system sends it a signal (SIGSEGV for an invalid memory access, SIGILL for an illegal instruction, SIGFPE for an arithmetic fault such as division by zero) and the debugger stops there with the CPU state intact. A reliable routine for reading a crash:

  1. Which instruction? x/i $pc (or => in disassemble) shows the faulting instruction.
  2. What address did it touch? For a memory access like mov eax, DWORD PTR [rdx+rcx*4], print the registers it uses (p/x $rdx, p/x $rcx) and work out the effective address. A fault address near 0 usually means a null pointer plus a small field offset. A huge, odd-looking address often means a corrupted pointer.
  3. How did we get here? backtrace lists the chain of callers, each recovered from the return addresses on the stack.
  4. Where did the bad value come from? Look at nearby instructions, or set a watchpoint on the variable and run again.

The following demo applies the routine to a crashing toy program. It loads through a pointer in rbx, and rbx is still 0.

JS Reading a crash with the toy debugger

Press Run (or Ctrl + Enter) to see the console output here.

No output.

That reading, "pointer is 0 plus offset 4", is exactly what experienced developers do within seconds of seeing a null-pointer crash in a debugger.

Practical tips

  • Use display/i $pc in gdb, so the next instruction is printed after every step.
  • At -O0 the code follows your C line by line. At -O2 the compiler reorders, merges and removes things; the debugger may show variables as <optimized out>, and stepping jumps around. Debug logic at -O0, then look at the optimized build for performance questions.
  • layout asm and layout regs in gdb open a text interface showing code and registers together.
  • If a program takes a long time to reach the interesting part, set a breakpoint on the function (break name) or a conditional breakpoint (break name if x == 5) instead of stepping from the start.
  • core dumps (a snapshot of memory and registers written when a program crashes) let you inspect a crash after the fact with gdb ./prog core, when your system is configured to write them.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the difference between stepi and nexti in gdb?

    stepi enters called functions; nexti steps over a call so the whole callee runs as one step.
  2. 2. What does x/4xw $rsp display?

    The format is x/NFU: N=4 units, F=x for hex, U=w for 4-byte words, starting at the address in rsp.
  3. 3. A crash shows mov eax, [rdx+8] with rdx = 0. What is the most likely cause?

    A fault address of 0 + 8 is a tiny address in the unmapped null page, which is the signature of a null pointer plus a field offset.
  4. 4. How does a debugger implement a software breakpoint on x86?

    int3 (0xCC) raises a trap that the debugger catches; it restores the original byte when continuing.

Summary

  • Compile with -g -O0 to learn; debuggers work on any binary, with or without symbols.
  • disassemble, stepi, nexti, info registers and x/NFU address are the core instruction-level commands (lldb has equivalents).
  • => marks the instruction about to run at rip; remember little-endian when reading raw bytes.
  • A software breakpoint is a trap instruction (int3) patched into the code; a hardware watchpoint stops when memory changes.
  • To read a crash: find the faulting instruction, compute the address it touched, check the backtrace, then trace where the bad value came from.

Next: System Calls and Talking to the OS, where your code asks the kernel to do real work.