Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 40 ยท Systems and Performance

Security: Buffer Overflows and Defenses

See how a buffer overflow overwrites a return address at the instruction level, then how canaries, NX, ASLR and constant-time code defend. Safe toy simulations.

Many of the worst security bugs in history come from one small mismatch: the CPU treats memory as a flat array of bytes, while the programmer thinks in arrays, strings and variables. Write a few bytes too many into an array on the stack and you are no longer writing into your variable, you are writing into whatever sits next to it, including the return address that ret will jump to. Assembly is where this becomes obvious, because there is nothing between you and the stack.

This lesson explains the mechanism and the standard defenses (stack canaries, non-executable memory, address randomization) and finishes with code that must not leak secrets through timing. Everything runs on the toy CPU as a simulation: the toy has no real addresses, no operating system and no shellcode, and nothing here attacks a real system. The aim is to understand why code is written defensively, not to give you a recipe against real software.

What is next to your buffer

Consider a C function with a local array and an unchecked copy:

C
void greet(const char *name) {
    char buf[16];
    strcpy(buf, name);     /* copies until the terminating zero, however long name is */
    puts(buf);
}

buf lives in the function's stack frame (Stack Frames, Prologues and Epilogues). The stack grows toward lower addresses, but an array is filled toward higher addresses, so an overlong copy runs up the frame, away from the buffer and toward the older data.

flowchart TB
  A["Return address (the caller's next instruction)"]
  B["Saved rbp"]
  C["Local variables"]
  D["buf[0] ... buf[15]  (written upward)"]
  D -->|"overflow climbs"| C
  C --> B
  B --> A

Higher addresses are at the top: the overflow starts in buf and climbs toward the return address.

When the function ends, ret pops whatever 8 bytes now sit in the return-address slot and jumps there. If an attacker chose those bytes, the attacker chose where the program continues. That is the whole idea of a stack smashing attack. Which address is useful, and how the attacker learns it, is what the defenses attack.

A toy stack smash

The toy program below calls vuln, which copies a number of words from an input area into a 4-word stack buffer and never checks the count. The caller then prints 1 ("returned normally"). Somewhere else in the program sits a function called secret that nothing ever calls and that prints 1337.

The input is just data placed in memory at address 0x2000. The "attacker" in the script reads the address (the instruction index) of secret from the assembler's label table, which stands in for the information leaks and guesswork real attackers need.

JS Overflow the buffer, redirect the return

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Four words fit exactly and the program returns normally. A fifth word lands in the slot where call stored the return address, so the next ret pops it. With junk, the CPU jumps to an address with no code and the run ends (a real machine would crash with a segmentation fault, the usual symptom of a smashed stack). With the address of secret, the program returns into a function nobody called. Nothing in the hardware stopped it: ret just obeys the stack.

On a real machine the target is rarely a convenient secret function. Historically the attacker also wrote machine code into the buffer and returned into the buffer itself; the defenses below grew up to prevent exactly that and the more indirect tricks that followed.

Defense 1: stack canaries

A stack canary is a secret value the compiler places between the locals and the return address. The function's prologue stores it; the epilogue checks it before ret. An overflow that reaches the return address must first run over the canary, so a changed canary means "stop, the frame is corrupt".

Here is what GCC typically emits for greet when -fstack-protector-strong is on (the default in many Linux distributions). The fs:0x28 operand reads the canary from per-thread storage that the C library filled with a random value at program start:

NASM
greet:
    push  rbp
    mov   rbp, rsp
    sub   rsp, 32
    mov   QWORD PTR [rbp-24], rdi
    mov   rax, QWORD PTR fs:0x28      ; load this process's canary
    mov   QWORD PTR [rbp-8], rax      ; park it just below the saved rbp
    xor   eax, eax
    ; ... strcpy and puts ...
    mov   rdx, QWORD PTR [rbp-8]
    sub   rdx, QWORD PTR fs:0x28      ; unchanged? the result is zero
    je    .ok
    call  __stack_chk_fail            ; corrupt frame: abort the process
.ok:
    leave
    ret

The toy version below uses a fixed value, 0xC0FFEE, because it has no random number source. A real canary is random per run, and on glibc its lowest byte is zero so that a string copy, which stops at a zero byte, cannot easily write past it. The attacker has two options: guess the value (hopeless for 7 random bytes) or leak it first through a separate bug and then write the correct value back. The third scenario shows that second case: canaries raise the cost, they do not make a leaky program safe.

JS The same overflow against a canary

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Defense 2: non-executable memory (NX, DEP)

The classic attack wrote code into the buffer and returned into it. Modern CPUs and operating systems stop that by giving every page a permission set: read, write, execute. The stack and heap are mapped writable but not executable (called NX or XD on x86, XN on ARM, DEP on Windows), and code pages are executable but not writable. This rule is often written W^X: no page is both. Returning into the buffer now faults at once, which is the same outcome as our "junk address" run. Our toy CPU enforces a version of this by design: it keeps instructions apart from data, so you could never jump into the input words.

Defense 3: address randomization (ASLR)

If the attacker's trick depends on knowing an address, hide the addresses. Address space layout randomization loads the stack, heap, libraries and (for position-independent executables, built with -fPIE -pie) the program itself at random locations on every run. The attacker's hard-coded address is then wrong for almost every run, which usually crashes the program instead of taking it over.

Return-oriented programming: reusing existing code

NX stops new code from running but not existing code. If the attacker cannot inject instructions, they can chain together short snippets that already exist in the program or its libraries. Each snippet, called a gadget, ends in ret. The attacker's overflow writes a long list of gadget addresses, and each ret hands control to the next one. The stack becomes the program counter. This is return-oriented programming (ROP).

flowchart LR
  S["overflowed stack:<br/>gadget_pop, 42, gadget_out, ..."] --> G1["gadget_pop:<br/>pop rax, ret"]
  G1 --> G2["gadget_out:<br/>out rax, ret"]
  G2 --> G3["next gadget ..."]

Each gadget ends with ret, so ret reads the next address from the attacker-controlled stack.

The toy has two tiny gadgets after the program's end: pop rax; ret and out rax; ret. The overflow writes a chain, and the program prints numbers it was never written to print. Then we rerun the same attack with the program "loaded at a different place" (we slide it by prepending harmless instructions), which is what ASLR does to the addresses.

JS A ROP chain, and what ASLR does to it

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With no slide the chain prints 42 and 7. Once the code moves, the attacker's addresses point into the middle of other code. In this toy the result is nothing, a wrong number or a hang; on a real system it is almost always a crash. Randomization is a probabilistic defense: it fails if the attacker can leak one address, because everything in a module moves together, so one leaked pointer reveals all its other addresses.

Defenses stack. Modern toolchains and CPUs add more, such as hardware-enforced shadow stacks (Intel CET) and pointer authentication on newer ARM cores, which both aim at protecting return addresses directly. Treat them all as a safety net. The real fix is not writing past a buffer in the first place: bounds-checked copies (strncpy done carefully, snprintf, memcpy with a verified length), memory-safe languages, compiler hardening (-D_FORTIFY_SOURCE=2, -fstack-protector-strong) and tools like AddressSanitizer in testing.

Constant-time code: leaking through time

Not every attack changes control flow. If a function takes longer when your guess is closer to a secret, you can recover the secret by measuring. The classic bug is a comparison that returns at the first mismatch.

The script below counts the work instead of timing it (so the demo is exact, not noisy): earlyExit returns as soon as a character differs, so the step count tells an attacker how many leading characters were right. The attacker guesses one position at a time, keeping the character that produced the most steps.

JS Recovering a secret from step counts

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The attacker needs at most 7 positions times 36 characters, about 250 tries, instead of 36 to the power 7. The constant-time version does the same work for every input: it ORs the differences together and decides only at the end. Real constant-time code in assembly follows the same rule, xor the bytes, or the results into an accumulator, and test once at the end. It also avoids secret-dependent branches, secret-dependent array indexes (cache timing) and variable-time instructions such as division on some CPUs. Use the library's vetted function (such as CRYPTO_memcmp, sodium_memcmp or crypto.timingSafeEqual), because compilers may rewrite clever code back into an early-exit loop.

Learning safely

The right place to practice is somewhere you own: your own machine, a virtual machine you can break and restore, or a legal practice environment. Capture-the-flag competitions and online wargames publish deliberately vulnerable programs for exactly this purpose. Compile your own tiny vulnerable C program with the protections turned off (-fno-stack-protector -z execstack -no-pie and disable ASLR in your VM), step through it with the debugger from Debugging at the Instruction Level, and watch the frame change. Never test on systems you do not own or have written permission to test.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does an overflowing array on the stack threaten the return address?

    Writes into an array climb toward higher addresses, and the frame holds locals, then saved registers, then the return address. A long enough write overwrites it, and ret jumps wherever it points.
  2. 2. What does a stack canary do?

    The prologue stores a (random, in real systems) value between the locals and the return address and the epilogue verifies it. An overflow that reaches the return address has to alter it first.
  3. 3. NX (DEP, W^X) mainly protects against...

    Pages are either writable or executable, never both, so bytes an attacker writes into the stack or heap cannot be run. It does not stop reuse of existing code, which is why ROP exists.
  4. 4. How does return-oriented programming get around NX?

    No new code is injected. The attacker places addresses of existing gadgets on the stack, and each gadget's ret transfers to the next.
  5. 5. Why should a secret comparison avoid returning at the first mismatch?

    If the work done depends on the secret, an attacker can learn the secret one position at a time from the timing. Constant-time code does the same work for every input.

Summary

  • A buffer overflow writes past an array into neighboring stack data; reaching the saved return address lets the attacker choose where ret goes.
  • Stack canaries place a random value before the return address and check it on return; they can be bypassed if the value leaks.
  • NX / DEP (W^X) makes writable memory non-executable, so injected bytes cannot run; ROP reuses existing gadgets ending in ret instead.
  • ASLR randomizes addresses so that hard-coded targets are wrong; one leaked pointer can undo it for a whole module.
  • Defenses form layers around the real fix: bounds checking, safe languages, compiler hardening and testing with sanitizers.
  • Secret-dependent time, branches or memory indexes leak information; constant-time code does identical work for every input.
  • Practice only on systems you own, in a throwaway virtual machine or a legal practice environment.

Next: put the whole toolbox to work, starting with Project: A Stack-Based Calculator.