Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 40 ยท The Stack and Functions

Calling Conventions: System V, Windows x64 and AAPCS64

Compare the System V, Windows x64 and ARM64 calling conventions: argument registers, return values, caller- vs callee-saved registers, shadow space and variadics.

call and ret get you to a function and back. They say nothing about how to hand the function its arguments, where the result will be, or which registers the function may wreck. If every compiler decided those details for itself, code from different compilers, libraries written in different languages, and the operating system could never call one another. A calling convention is the written agreement that fixes all of it, and it is part of a platform's ABI (application binary interface).

This lesson compares the three conventions you are most likely to meet: System V AMD64 (Linux, macOS on Intel, BSDs), Microsoft x64 (Windows) and AAPCS64 (the standard ARM64 convention, used by Linux and, with small variations, Apple platforms and Windows on ARM). It follows CALL and RET, where we used rdi and rax by informal agreement. Details below are drawn from the published conventions as widely documented; for anything critical, check the current ABI document for your platform.

What a convention decides

Every calling convention answers the same six questions:

  1. In which registers (and then where on the stack) do the arguments go?
  2. Where does the return value come back?
  3. Which registers must a function preserve (callee-saved), and which may it freely overwrite (caller-saved)?
  4. Who removes the arguments from the stack, and how much stack must the caller reserve?
  5. How must the stack be aligned at a call?
  6. How do functions with a variable number of arguments, such as printf, work?
flowchart LR
  A["Caller saves caller-saved<br/>registers it still needs"] --> B["Caller puts arguments<br/>in registers or on the stack"]
  B --> C["call"]
  C --> D["Callee saves the callee-saved<br/>registers it will use"]
  D --> E["Callee runs, puts result in the<br/>return register, restores registers"]
  E --> F["ret"]
  F --> G["Caller reads the result"]

The agreement splits the work: the callee protects its callee-saved registers, the caller protects everything else it cares about.

The three conventions at a glance

System V AMD64 Windows x64 AAPCS64 (ARM64)
Integer/pointer arguments rdi, rsi, rdx, rcx, r8, r9 rcx, rdx, r8, r9 x0 to x7
Floating-point arguments xmm0 to xmm7 xmm0 to xmm3 (same position as integers) v0 to v7 (d0 to d7)
Further arguments on the stack, 8-byte slots on the stack, after the shadow space on the stack
Return value rax (and rdx for 16 bytes), xmm0 for floats rax, xmm0 for floats x0 (and x1), v0 for floats
Callee-saved (preserved) rbx, rbp, r12 to r15 rbx, rbp, rdi, rsi, r12 to r15, xmm6 to xmm15 x19 to x28, x29, low 64 bits of v8 to v15
Caller-saved (free to clobber) rax, rcx, rdx, rsi, rdi, r8 to r11, all xmm rax, rcx, rdx, r8 to r11, xmm0 to xmm5 x0 to x15 (x16 and x17 are for linkers), v0 to v7, v16 to v31
Return address pushed on the stack by call pushed on the stack by call in x30 (link register)
Stack alignment 16 bytes at call 16 bytes at call sp 16 bytes aligned at all times it is used
Special 128-byte red zone 32-byte shadow space, no red zone x8 holds the address for large struct results

Some patterns stand out. System V and AAPCS64 use separate counters for integer and floating-point arguments: the third argument of f(int, double, int) is the second integer register. Windows x64 is positional: argument number n uses the n-th slot whatever its type, so the same call would put the third argument in r8. And on ARM64 the first eight integer arguments fit in registers, so far fewer calls touch the stack at all.

Note the register x18, reserved as a platform register on Apple platforms and on Windows (do not use it), and that on Windows, rdi and rsi are callee-saved while on System V they carry arguments. Code that assumes one convention while running under the other breaks in confusing ways.

Where do the arguments go?

The JavaScript below is a small model of the three conventions for scalar arguments (integers and doubles only, no structs). Give it a signature and it tells you the register or stack slot of each argument. Stack addresses are shown as seen by the callee at its first instruction: on x86-64 the return address sits at [rsp], so the first stack argument of System V is at [rsp+8]; on Windows the 32-byte shadow space sits above the return address, so the fifth argument is at [rsp+40]; on ARM64 nothing was pushed, so the first stack argument is at [sp].

JS Where do the arguments of f(int, double, int, ...) go?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Compare the columns. In System V the integer registers run out after six and the seventh integer argument (argument 8, because of the doubles in between) goes to the stack. The double in position 9 still finds a free xmm register. Windows ran out of registers after four positions, so everything from argument 5 on is on the stack, including the double. ARM64 fits all but the doubles in x0 to x7.

The same function, three conventions

Here is long add3(long a, long b, long c), read-only and typical of optimised output. The arguments sit in different registers but the structure is the same:

NASM
; System V AMD64
add3:   lea   rax, [rdi+rsi]     ; a + b
        add   rax, rdx           ; + c
        ret

; Windows x64
add3:   lea   rax, [rcx+rdx]
        add   rax, r8
        ret

; AAPCS64 (ARM64)
add3:   add   x0, x0, x1
        add   x0, x0, x2
        ret

Caller-saved versus callee-saved

Registers are split into two groups. Caller-saved (also called volatile) registers can be overwritten by any function; if the caller still needs the value after a call, the caller must save it before the call. Callee-saved (non-volatile) registers must have the same value when the function returns as when it was called; a function that wants to use one must save it first (usually with push or stp) and restore it before ret.

The split is a compromise. A function that needs a few scratch registers can use the caller-saved ones for free; a long-lived value such as a loop counter in a function that makes many calls belongs in a callee-saved register, saved once instead of around every call.

In the toy CPU, we declare rbx callee-saved (as in System V). The program below shows what happens when the callee ignores the rule. The caller keeps 5 in rbx across a call and expects double_it(3) + 5 = 11.

JS A callee that breaks the contract

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The buggy version prints 12 instead of 11, because rbx came back as 6. Nothing crashes; the program is just silently wrong, which is why violating a convention is so hard to debug. The careful version prints 11.

Checking a function against the rules

The next TryIt is a tiny convention checker: you describe which registers a routine writes and which it restores, and the checker reports violations under each convention. The same routine can be legal under one convention and illegal under another.

JS Is this routine allowed to write these registers?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

For System V only rbx is a violation: rsi is an argument register, so it is caller-saved. Under Windows the same routine also breaks the rules for rsi and xmm6, since both are non-volatile there. On ARM64, x20 and d8 are in the preserved set, and x30 (the link register) must be back to its entry value when the function returns, or ret will jump to the wrong place; here only x20 and d8 are reported because x19 and x30 were restored. The d8 entry reflects that only the low 64 bits of v8 to v15 are preserved.

Shadow space, stack alignment and variadic functions

Shadow space (Windows x64). The caller always reserves 32 bytes on the stack just above the return address, even for functions with fewer than four arguments. The callee may use this "home" area to spill its register arguments. Forgetting to reserve it is a classic bug when writing Windows assembly, because the callee then overwrites the caller's data. System V has no equivalent.

Alignment. For both x86-64 conventions rsp must be 16-byte aligned before the call, as discussed in The Stack; ARM64 requires an aligned sp whenever it is used for memory access.

Variadic functions. A function like printf(const char *fmt, ...) does not know how many arguments it will get. System V adds a rule: the caller must set al to an upper bound on the number of vector registers used, so a variadic callee can decide whether to save the xmm registers. So a call such as printf("%d %f\n", 1, 2.0) loads the format into rdi, the integer into rsi, the double into xmm0, and sets eax to 1 before the call. Windows x64 requires that floating-point variadic arguments are also copied to the matching integer registers. AAPCS64 itself passes variadic arguments like ordinary ones, but Apple's ARM64 convention differs: it passes the variadic part on the stack. That is one reason the same C code can need different generated code on Linux and macOS even for the same processor.

When a function pointer is called with the wrong signature, or a variadic function is called without a prototype, the compiler may put arguments where the callee does not look. This is undefined behavior in C, and many "mysterious" crashes come from it.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Under System V AMD64, which registers carry the first two integer arguments?

    System V uses rdi, rsi, rdx, rcx, r8, r9 in that order. rcx and rdx lead the Windows x64 list, and x0 and x1 are ARM64 registers.
  2. 2. What does it mean that a register is callee-saved?

    A callee-saved register is the callee's responsibility: use it only if you restore the original value before ret. Caller-saved registers, in contrast, may be freely overwritten.
  3. 3. What is the Windows x64 shadow space?

    Every Windows x64 call reserves 32 bytes, enough for the four register arguments, whether or not the callee needs them. System V has no shadow space.
  4. 4. For f(int a, double b, int c) on System V and on Windows x64, where does c go?

    System V counts integers and floats separately, so c is the second integer argument (rsi). Windows is positional, so c is the third argument, which uses r8.
  5. 5. Where does the return address live on ARM64 right after a bl instruction?

    bl stores the return address in x30 (lr). A function that makes further calls must save x30 itself, normally on the stack.

Summary

  • A calling convention (part of the ABI) fixes where arguments and results live, which registers must be preserved, and how the stack is aligned.
  • System V AMD64 passes six integer arguments in rdi, rsi, rdx, rcx, r8, r9 and floats in xmm0 to xmm7; Windows x64 passes four, positionally, in rcx, rdx, r8, r9; ARM64 passes eight in x0 to x7.
  • Results come back in rax or x0 (floats in xmm0 or v0).
  • Callee-saved registers must be restored by the function that uses them; caller-saved ones are free scratch, but the caller must save them if it needs them after a call.
  • Windows x64 reserves a 32-byte shadow space; System V has a 128-byte red zone; ARM64 keeps the return address in x30.
  • Variadic calls have extra rules (System V sets al, Windows duplicates floats in integer registers, Apple ARM64 puts variadic arguments on the stack).

Next: see how a function builds its own working area on top of these rules in Stack Frames, Prologues and Epilogues.