Skip to content
Tutorials Center
My learning

Topics

Lesson 57 of 95 ยท Processes and the System

strace, ltrace, lsof, perf and eBPF Intro

Find out what a Linux process is really doing: trace system calls with strace, list open files with lsof, profile with perf and meet eBPF tools.

Sooner or later a program on your server misbehaves and its own logs say nothing useful. It hangs, it is slow, it "cannot open a file" without telling you which one, or a disk is full and no file explains why. At that point you stop reading the program's source and start observing the running process from the outside. Linux offers an unusually rich set of tools for this, and most of them answer one of four questions: what is it asking the kernel for? (system calls), which files and sockets does it hold? (open files), where does the CPU time go? (profiling), and what is happening across the whole machine? (tracing with eBPF).

This lesson tours strace, ltrace, lsof, perf and the eBPF tools, and shows the /proc files that all of them read. Our sandbox does not have strace, lsof or perf, and tracing another process needs ptrace rights, so the real tool sessions are read-only. The runnable examples rebuild the key ideas from /proc, which you can read without any special tool. It assumes you know Processes and Users, Groups and sudo.

The layers you can look at

A running program talks to the system on several levels. Each tool in this lesson attaches to a different one:

flowchart TD
    P["Your program"] -->|"calls library functions<br/>(printf, malloc, getaddrinfo)"| L["Shared libraries<br/>(libc, libssl, ...)"]
    L -->|"system calls<br/>(openat, read, write, connect)"| K["Linux kernel"]
    P -.->|"ltrace shows this layer"| L
    P -.->|"strace shows this layer"| K
    K --> H["Files, sockets, disks, network, CPU"]
    H -.->|"lsof lists what is open"| K
    K -.->|"perf and eBPF watch kernel events"| K

A program calls libraries, libraries make system calls, the kernel does the real work. Each tool observes one of these boundaries.

Almost everything a program can do to the outside world, such as opening a file, reading from a socket, creating a process or sleeping, goes through a system call. That is why strace is so powerful: it shows the conversation between the program and the kernel, and it does not need the source code, debug symbols or even a recompile.

strace: watch system calls

strace runs a command (or attaches to a running process) and prints every system call with its arguments and return value. A short session looks like this (read-only; the exact lines vary by system):

CONSOLE
$ strace -e trace=openat cat /etc/missing.conf
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/missing.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
cat: /etc/missing.conf: No such file or directory
+++ exited with 1 +++

Each line is name(arguments) = result. A negative result is followed by the error name and its text. The third line is the answer you usually look for: the program asked for /etc/missing.conf, and the kernel said ENOENT. If a program "cannot find its configuration", strace tells you the exact path it tried, which is often a different one than you expected.

The options you will use most:

Option Effect
-f follow child processes and threads (essential for servers and shell scripts)
-p PID attach to a running process; needs the same user or root
-e trace=file only calls that take a path (openat, stat, unlink ...); other classes: network, process, memory, signal
-e trace=openat,read an explicit list of calls
-o out.txt write the trace to a file instead of stderr
-tt wall-clock timestamp on every line
-T time spent inside each call, in angle brackets
-s 200 show up to 200 bytes of string arguments (default is 32)
-c do not print calls; print a summary table at the end
-y show the path behind each file descriptor number

The summary mode answers "where does it spend its time in the kernel?":

CONSOLE
$ strace -c -f ./slow-report
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 71.20    1.904312          31     61244           read
 22.87    0.611904          10     61200           write
  3.10    0.082951        2072        40        12 openat
 ...

Here read and write account for 94% of the kernel time, and the call counts show the program moves data in tiny pieces: roughly 61,000 calls for a modest file. Switching to bigger buffers would help far more than any amount of CPU tuning.

Four classic findings:

  1. A hang. strace -p PID shows the process stuck in futex(...), poll(...), read(0, ...) or connect(...). The last call tells you what it is waiting for: input, a lock or a remote host.
  2. A missing file or permission. ENOENT and EACCES results on openat identify the path.
  3. A slow start. With -tt -T you see long gaps, often DNS lookups (connect to port 53) or timeouts.
  4. A mystery write. -e trace=file shows who deletes or rewrites a file.

strace slows the traced process, often by a factor of 10 or more, because every call stops twice for the tracer. Do not leave it attached to a busy production process longer than needed, and remember that traces contain whatever data the program read and wrote, including passwords. Treat the output file as sensitive.

ltrace: watch library calls

ltrace does the same one level up. It shows calls into shared libraries, such as malloc, strlen, fopen or getaddrinfo, which strace cannot see because they do not enter the kernel:

CONSOLE
$ ltrace -e 'str*' ./greet Maria
strlen("Maria")                      = 5
strcpy(0x7ffc..., "Maria")           = 0x7ffc...

ltrace works on dynamically linked programs only. It is rarely installed by default, and on modern systems it can be fragile, so reach for strace first and ltrace when you want to see which library function did something. To see which libraries a program loads at all, no tracer is needed:

CONSOLE
$ ldd /bin/ls                        # libraries resolved at run time
$ readelf -d /bin/ls | grep NEEDED   # libraries it declares
$ nm -D /bin/ls | grep ' U '         # library functions it imports (U = undefined here)

lsof: list open files

On Unix "everything is a file": regular files, directories, pipes, sockets and devices are all opened through file descriptors. lsof ("list open files") prints who holds what:

CONSOLE
$ lsof -p 4242                       # everything process 4242 has open
$ lsof /var/log/app.log              # who has this file open?
$ lsof +D /srv/data                  # every open file under a directory
$ lsof -i :8080                      # who listens on or uses port 8080?
$ lsof -iTCP -sTCP:LISTEN -P -n      # all listening TCP ports, numeric
$ lsof -u deploy                     # everything a user has open
$ lsof +L1                           # open files whose link count is 0 (deleted)

The last command finds a very common mystery. A file that is deleted while a process still has it open keeps using disk space until the process closes it or exits. df says the disk is full, du cannot find the space, and lsof +L1 shows a log file marked (deleted) that a daemon still writes to. The fix is to restart the daemon, or to truncate the file through the descriptor (: > /proc/PID/fd/N) without a restart.

What lsof reads: /proc

lsof and ps get their data from /proc, a virtual filesystem where each process has a directory /proc/PID/. Several files in there are enough for a home-made lsof:

Path Content
/proc/PID/fd/ one symbolic link per open file descriptor, pointing at the file, socket:[inode] or pipe:[inode]
/proc/PID/cmdline the command line, arguments separated by NUL bytes
/proc/PID/exe, cwd links to the running executable and the current directory
/proc/PID/status readable state, parent, threads, memory
/proc/PID/environ the environment (readable only by the owner)
/proc/PID/maps the memory map, including loaded libraries

You can read the files of your own processes without any privilege. This script starts a background process that inherits two files, then lists its descriptors the way lsof -p would:

Bash A tiny lsof built from /proc

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

You will see descriptors 0, 1 and 2 (the standard streams) followed by 3 and 4, the two files we opened in the parent. Children inherit open descriptors, which is exactly how shells wire pipes and redirections (see Pipes and Streams). The sandbox connects 1 and 2 to sockets and 0 to /dev/null; in your terminal they would be /dev/pts/N.

The deleted-file mystery, reproduced

Here is the disk-full puzzle in miniature. A "daemon" (our shell) holds a log open, someone removes the log, and the space stays allocated. The /proc link still names the file with a (deleted) suffix, and you can even read the data back through it:

Bash A deleted file that is still open

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

On a real server the same recipe saves data from an accidental rm of a file that a running process still has open: copy it back with cp /proc/PID/fd/N /safe/place. It stops working the moment the process closes the file.

perf: where does the CPU time go?

System-call tracing tells you what a program asks of the kernel. When a program is simply busy (the CPU is at 100%), you need a profiler. perf is the profiler that ships with the kernel source and is available as linux-tools or perf in most distributions. It samples the CPU many times per second and records which function was running:

CONSOLE
$ perf stat ./job                    # counters: cycles, instructions, cache misses, time
$ perf top                           # live view, like top but for functions
$ perf record -g ./job               # sample with call stacks into perf.data
$ perf report                        # browse the result interactively
$ perf record -F 99 -g -p 4242 -- sleep 30    # attach to a running process for 30 s

A typical perf report ranks functions by their share of samples:

TEXT
Overhead  Command  Shared Object   Symbol
  41.8%   job      job             [.] compute_checksum
  18.3%   job      libc.so.6       [.] __memcpy_avx_unaligned
  12.1%   job      [kernel]        [k] copy_user_enhanced_fast_string

[.] marks user-space symbols and [k] kernel ones. The usual path is: find the biggest bar, look at its callers (-g), fix it, measure again. Many teams turn perf record -g output into flame graphs, where the width of each box is the time spent in that function and its callees. Containers and some cloud kernels restrict perf through kernel.perf_event_paranoid; you may need root or to lower that setting (see Boot Process, GRUB, Kernel Modules and sysctl).

eBPF: tracing without stopping the process

strace is a blunt instrument: it halts the process at every call. eBPF (extended Berkeley Packet Filter) is a way to run small, verified programs inside the kernel and attach them to events such as system calls, function entries, network packets and disk requests. The kernel checks each program for safety before loading it, so it cannot crash the machine, and the overhead is low enough for production.

You rarely write eBPF by hand. You use ready-made tools:

Tool Answers
execsnoop which new processes start, with arguments (catches short-lived ones)
opensnoop which files are opened, by whom, and which fail
biolatency a histogram of disk I/O latency
tcpconnect, tcplife outgoing TCP connections and their duration
bpftrace a small scripting language for custom one-liners

For example, a bpftrace one-liner that counts system calls per program for as long as it runs (read-only, needs root):

CONSOLE
$ sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
^C
@[sshd]: 214
@[nginx]: 5890
@[job]: 91037

The tools usually come as bpfcc-tools or bpftrace packages and need a reasonably recent kernel and root. They also keep working where strace is too slow or ptrace is blocked, and they are the foundation of many observability products.

Choosing the right tool

Symptom First tool
"It cannot find / open a file" strace -f -e trace=file
"It hangs" strace -p PID, then lsof -p PID
"Disk is full but I cannot find the files" lsof +L1
"Which process owns port 8080?" lsof -i :8080 (or ss -tlnp)
"The CPU is at 100%" perf top, perf record -g
"Everything is slow somewhere" biolatency, execsnoop, opensnoop
"Which libraries does it use?" ldd, readelf -d, ltrace

One more tool needs no installation: the shell itself. bash -x script.sh prints every command before running it, and setting PS4 adds line numbers; see Debugging Scripts.

Finally, look at a binary's declared dependencies. This works without any tracer and is a quick way to understand what a program is built on:

Bash Which libraries does ls need?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The output tells you that ls links against the C library and a SELinux library, and that it imports opendir and readdir to list directories. If you ran ltrace ls, those are the calls you would see.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A program prints only cannot open configuration. Which command shows the exact file path it tried to open?

    strace shows the system calls. With -e trace=file you see each openat or stat with its path and the error, for example ENOENT, so you learn which path was tried.
  2. 2. df says a disk is 100% full, but du finds far less data. Which command is most likely to reveal the cause?

    A file deleted while a process still has it open keeps its disk space. lsof +L1 lists open files with link count zero, shown as (deleted).
  3. 3. Why is strace -p on a busy production server something to use briefly?

    ptrace-based tracing interrupts the process twice per system call. That can slow it by an order of magnitude. eBPF-based tools have far lower overhead.
  4. 4. What does ltrace show that strace does not?

    ltrace follows calls between the program and its dynamic libraries. Many of those never reach the kernel, so strace cannot see them.
  5. 5. Where does lsof get the list of open files of a process on Linux?

    /proc is a virtual filesystem generated by the kernel. Each entry in /proc/PID/fd is a link to the open file, socket or pipe, and you can read it with ls -l or readlink yourself.

Summary

  • Observe a misbehaving process from outside: system calls with strace, library calls with ltrace, open files with lsof, CPU time with perf, kernel events with eBPF tools.
  • strace -f -e trace=file finds missing files and permission errors; -c summarizes time per call; -p PID attaches to a running process; it slows the target noticeably.
  • lsof -p PID, lsof -i :PORT and lsof +L1 answer "what is open?", "who uses this port?" and "which deleted file still eats my disk?".
  • /proc/PID/fd, cmdline, exe, status and maps hold the raw facts, readable for your own processes without special tools; a deleted but open file can be copied back through /proc/PID/fd/N.
  • perf stat, perf top and perf record -g with perf report rank where the CPU goes; flame graphs visualize the same data.
  • eBPF programs run safely in the kernel with low overhead; execsnoop, opensnoop, biolatency and bpftrace are the usual entry points and need root.
  • Traces may contain secrets, so protect the files you save.

Next: Expansion Order