Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 40 ยท Systems and Performance

System Calls and Talking to the OS

How assembly asks the operating system for help: the syscall instruction, Linux x86-64 and ARM64 numbers, write and exit, kernel vs user mode, with demos.

A program running on the CPU can add numbers, move memory and jump around, but it cannot, by itself, print to the screen, open a file, or send a network packet. Those jobs belong to the operating system kernel, which owns the hardware. When your program needs one of them, it makes a system call (often shortened to syscall): a controlled request to the kernel.

In this lesson you learn what happens at the instruction level during a syscall, how the Linux x86-64 and ARM64 conventions pass the request number and arguments, how to write a real "hello, world" with nothing but write and exit, and why Windows does things differently. The real code is read-only in most places, and a runnable toy kernel lets you practise the convention; one example calls the real kernel through the site's x86-64 Linux runner.

User mode and kernel mode

The CPU runs code in different privilege levels. Ordinary programs run in user mode and are not allowed to touch hardware or other processes' memory directly. The kernel runs in kernel mode, where everything is allowed. (On x86 these are "ring 3" and "ring 0"; on ARM64 they are "exception levels" EL0 and EL1.)

A program cannot just jump into kernel code: that would defeat the protection. Instead the CPU provides a special instruction (syscall on x86-64, svc on ARM64) that switches to kernel mode and jumps to one fixed entry point that the kernel chose at boot. The kernel inspects the request, does the work, and returns to the program with a result.

sequenceDiagram
  participant P as Your program (user mode)
  participant K as Kernel (kernel mode)
  P->>P: put number in rax, arguments in rdi, rsi, rdx
  P->>K: syscall instruction (switch to kernel mode)
  K->>K: look up number, check arguments, do the work
  K-->>P: result in rax, return to the next instruction

A syscall is a controlled doorway: one instruction in, one result out, with the kernel in charge in between.

A syscall is much more expensive than an ordinary function call (hundreds of cycles at least, more with security mitigations), which is why libraries buffer output instead of calling write for every single character.

The Linux x86-64 convention

On Linux x86-64 you load the syscall number into rax and up to six arguments into specific registers, then execute syscall:

Register Role
rax syscall number on entry; return value on exit
rdi argument 1
rsi argument 2
rdx argument 3
r10 argument 4 (note: not rcx)
r8 argument 5
r9 argument 6

The argument registers resemble the System V calling convention, with one difference: argument 4 is r10, because the syscall instruction itself overwrites rcx (it saves the return address there) and r11 (it saves the flags). All other registers are preserved across the call. On return, rax holds the result; a value between -4095 and -1 is a negated error number (-EBADF is -9). The C library's wrapper functions turn that into the familiar "return -1 and set errno".

A few Linux x86-64 syscall numbers (they differ per architecture, so never assume):

Number Name Arguments
0 read fd, buffer, count
1 write fd, buffer, count
2 open path, flags, mode
3 close fd
39 getpid none
60 exit status
231 exit_group status (ends all threads)

Standard file descriptors are 0 (input), 1 (output) and 2 (error output). Here is a complete program that prints a line and exits, written for the NASM assembler (how to build it):

ASM
section .data
msg:  db "Hello, syscalls!", 10
len:  equ $ - msg

section .text
global _start
_start:
  mov eax, 1          ; syscall number 1 = write
  mov edi, 1          ; arg 1: file descriptor 1 = stdout
  mov esi, msg        ; arg 2: address of the bytes
  mov edx, len        ; arg 3: how many bytes
  syscall             ; ask the kernel; rax = bytes written

  mov eax, 60         ; syscall number 60 = exit
  xor edi, edi        ; arg 1: exit status 0
  syscall             ; never returns

There is no C runtime here, no main, and no ret. The program starts at _start, and if it fell off the end, the CPU would execute whatever bytes came next. Ending with the exit syscall is mandatory. (Writing to a register with a 32-bit name such as eax also clears the upper half, which is why mov eax, 1 is enough; see the x86-64 registers.)

The ARM64 convention

ARM64 Linux uses a different instruction and different registers. The syscall number goes in x8, arguments in x0 to x5, the instruction is svc 0 (supervisor call), and the result comes back in x0. The numbers are also different, because Linux ARM64 uses a newer, cleaned-up table:

Number Name
63 read
64 write
56 openat
57 close
172 getpid
93 exit
94 exit_group

The same program for ARM64 Linux (GNU assembler):

ASM
        .section .data
msg:    .ascii "Hello, syscalls!\n"

        .section .text
        .globl _start
_start:
        mov     x0, #1          // fd 1 = stdout
        ldr     x1, =msg        // address of the text
        mov     x2, #17         // length in bytes
        mov     x8, #64         // syscall number 64 = write
        svc     #0

        mov     x0, #0          // exit status 0
        mov     x8, #93         // syscall number 93 = exit
        svc     #0

macOS on Apple Silicon follows yet another convention (the syscall number goes in x16 and the instruction is svc #0x80), and Apple discourages calling the kernel directly; programs go through the system library instead. The takeaway is that a syscall interface is defined per operating system and per architecture.

A toy kernel you can run

The registers-and-number convention is easiest to learn by using it. The stand-in below has a miniature CPU that understands mov and syscall, plus a tiny "kernel" with write, getpid and exit. The kernel returns the negative error codes you would see on Linux: -9 (EBADF, bad file descriptor) when you write to a descriptor that is not open, and -38 (ENOSYS, no such syscall) for a number it does not know. The numbers here are the real Linux x86-64 ones, but this is a model, not the real kernel.

JS A toy kernel and syscall instruction

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice that the write returns 17, the number of bytes (the length of Hello, syscalls! plus the newline), and that failures are ordinary negative numbers in rax. A real program must check them: write can legitimately write fewer bytes than requested, and a negative value means nothing was written.

Calling the real kernel from C

You do not need a standalone assembly file to make a raw syscall. C's syscall() function (from <unistd.h> and <sys/syscall.h>) takes the syscall number and arguments and does exactly what the toy did. Inline assembly with the syscall instruction does it without any wrapper, using constraints to place the values in rax, rdi, rsi and rdx, and declaring that rcx, r11 and memory are changed.

The program below runs on the site's real x86-64 Linux runner. It makes the same write call three ways and then deliberately writes to a descriptor that is not open to show a real -EBADF.

C Real syscalls through libc and inline asm

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with scanf or fgets), it gets the next line from this box.

The calls print in order because the program flushes stdout before the first direct write and does not mix buffered output into the middle. Mixing printf with direct write calls is a classic source of out-of-order output, because printf buffers in user space while write goes straight to the kernel.

What about Windows?

Windows has system calls too, but Microsoft treats the numbers as private. They change between Windows versions, so no program should hard-code them. Instead, applications call documented functions in DLLs: for example WriteFile in kernel32.dll, which calls into ntdll.dll, which executes the actual syscall instruction with the correct number for that Windows build. Native Windows assembly programs therefore call library functions using the Windows x64 calling convention rather than invoking the kernel directly.

A good rule is to treat the library as the stable interface on every system: Linux guarantees its syscall numbers for each architecture, but Windows and macOS guarantee only their libraries.

Why syscalls matter

Almost everything useful goes through them: files, sockets, processes, memory mapping (mmap), time, and signals. Tools such as strace on Linux print every syscall a program makes, with arguments and results. For example, strace ./hello shows a write(1, "Hello, syscalls!\n", 17) = 17 line, which is a nice way to see the numbers and registers you just learned in action.

Understanding syscalls also explains security tools. A process sandbox (seccomp on Linux, for example) works by restricting which syscall numbers a program may use, and the kernel can refuse anything dangerous at that one narrow doorway.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. On Linux x86-64, which register holds the syscall number when you execute syscall?

    The number goes in rax, the arguments in rdi, rsi, rdx, r10, r8 and r9, and the result comes back in rax.
  2. 2. Why does the fourth syscall argument use r10 instead of rcx?

    syscall saves the return address in rcx and the flags in r11, so rcx cannot carry an argument.
  3. 3. After a Linux write syscall, rax contains -9. What happened?

    Small negative values are negated errno codes returned directly by the kernel; -9 is EBADF.
  4. 4. Which instruction and register carry the syscall number on Linux ARM64?

    ARM64 Linux uses svc #0 with the number in x8, arguments in x0 to x5 and the result in x0.
  5. 5. Why should Windows programs call library functions instead of using syscall numbers directly?

    Microsoft documents the DLL interface, not the raw numbers, which vary from build to build.

Summary

  • A system call is a controlled switch from user mode to kernel mode, made with syscall (x86-64) or svc (ARM64).
  • Linux x86-64: number in rax, arguments in rdi, rsi, rdx, r10, r8, r9, result in rax; rcx and r11 are clobbered.
  • Linux ARM64: number in x8, arguments in x0 to x5, result in x0; the numbers differ from x86-64 (write is 1 on x86-64, 64 on ARM64).
  • Results from -4095 to -1 are negated error numbers; always check them.
  • A freestanding program must end with the exit syscall; libraries wrap syscalls and buffer output for speed.
  • Windows and macOS keep raw syscall numbers private; use their libraries.

Next: Floating Point and SIMD, where one instruction works on several numbers at once.