Skip to content
Tutorials Center
My learning

Topics

Lesson 86 of 95 ยท System Administration

Boot Process, GRUB, Kernel Modules and sysctl

Follow the Linux boot from firmware and GRUB to the kernel and systemd, edit the kernel command line, manage modules with modprobe and tune the kernel with sysctl.

Press the power button and, a few seconds later, you get a login prompt. In between, a chain of small programs hands control to the next one: firmware, a bootloader, the kernel, an early temporary filesystem and finally the init system. When a server does not come up, the first question is always which link of the chain broke, and you can only answer it if you know the chain. Once the machine is running, two more skills decide how it behaves: loading kernel modules (drivers) on demand, and changing kernel tunables with sysctl.

This lesson follows the boot from firmware to systemd, shows what GRUB and the kernel command line do, explains lsmod, modprobe and the files in /etc/modprobe.d/, and teaches sysctl with its configuration files. It builds on Packages, Services and systemd and Processes. The sandbox is a container without a bootloader or module loading rights, so system-changing commands are read-only; the runnable examples parse the real formats (/proc/cmdline, /proc/modules, sysctl.d and modprobe.d files) with Bash.

From power button to login prompt

flowchart TD
    A["Power on"] --> B["Firmware: UEFI or BIOS<br/>hardware self-test, picks a boot device"]
    B --> C["Bootloader: GRUB<br/>from the EFI System Partition or MBR"]
    C --> D["Loads kernel (vmlinuz)<br/>and initramfs into memory"]
    D --> E["Kernel starts<br/>detects hardware, mounts initramfs"]
    E --> F["initramfs: loads storage drivers,<br/>unlocks disks, mounts the real root"]
    F --> G["Kernel starts PID 1: systemd"]
    G --> H["systemd reaches a target<br/>(multi-user.target or graphical.target)"]
    H --> I["Login prompt, services running"]

Every step is a separate program. If one fails, the machine stops there, and the message you see tells you which step it was.

Here is what each stage does:

  1. Firmware. Modern machines use UEFI; older ones use the legacy BIOS. It tests the hardware and looks for something to boot. With UEFI, small programs called EFI applications live as files on the EFI System Partition (ESP), a FAT filesystem usually mounted at /boot/efi. Secure Boot makes the firmware accept only bootloaders and kernels signed with a trusted key.
  2. Bootloader. On Linux it is almost always GRUB 2. It shows the boot menu, lets you edit the kernel command line, finds the kernel and the initramfs, loads both into memory and jumps into the kernel.
  3. Kernel. The files in /boot are named after the version: vmlinuz-6.8.0-45-generic (the compressed kernel) and initrd.img-6.8.0-45-generic or initramfs-...img. The kernel initialises memory, CPUs and devices, using the command line you gave it.
  4. initramfs. The kernel cannot read your root disk until it has the right driver, and the driver may live on that disk. The initial RAM filesystem is a small archive unpacked into memory that contains just enough tools and modules (disk controllers, RAID, LVM, encryption, network for NFS root) to find and mount the real root filesystem. Then it hands over.
  5. init (systemd). The kernel starts one user-space program as PID 1: systemd on nearly every distribution (see Packages, Services and systemd). It mounts the other filesystems and starts services in parallel until it reaches the default target.

You can look at the result of all this on a running machine:

CONSOLE
$ ls /boot
config-6.8.0-45-generic  grub  initrd.img-6.8.0-45-generic  vmlinuz-6.8.0-45-generic
$ uname -r                           # the running kernel's release
6.8.0-45-generic
$ cat /proc/cmdline                  # the exact command line it was started with
$ systemd-analyze                    # total boot time, split into firmware, loader, kernel, userspace
$ systemd-analyze blame | head       # slowest units first
$ systemd-analyze critical-chain     # the chain of units that decided the boot time
$ journalctl -b                      # all logs of this boot; -b -1 = the previous boot
$ dmesg | less                       # the kernel's own ring buffer of messages

GRUB and the kernel command line

GRUB's menu is generated from two places. The file you edit is /etc/default/grub, plus scripts in /etc/grub.d/. The real configuration, /boot/grub/grub.cfg (or /boot/grub2/grub.cfg), is generated: never edit it by hand, because the next kernel update overwrites it.

TEXT
# /etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX="console=ttyS0,115200"

After a change you regenerate the file with sudo update-grub (Debian, Ubuntu) or sudo grub2-mkconfig -o /boot/grub2/grub.cfg (RHEL family; on UEFI systems check the path your distribution documents). GRUB_CMDLINE_LINUX_DEFAULT applies to normal boot entries, GRUB_CMDLINE_LINUX to all entries, including recovery ones.

The kernel command line is a plain string of options that configure the kernel and, indirectly, the early user space. Words with = are parameters; words without are flags. Some you should recognise:

Parameter Meaning
root=UUID=... which device is the root filesystem
ro mount the root read-only at first (init remounts it read-write after checking)
quiet, splash hide boot messages, show a graphical splash
console=ttyS0,115200 send kernel messages to a serial console (can appear several times)
systemd.unit=rescue.target boot into single-user rescue mode (or emergency.target)
init=/bin/bash run a shell as PID 1 instead of systemd, for password recovery
nomodeset do not load graphics drivers (fixes a black screen)
intel_iommu=on enable a hardware feature, usually for virtualization
modprobe.blacklist=nouveau keep a module from loading

To change the line for one boot, highlight the entry in the GRUB menu, press e, edit the line that starts with linux, and press Ctrl+X. Anyone with console access can do that, which is why physical security and a GRUB password matter. The running kernel's line is in /proc/cmdline. This script splits a sample into parameters, flags and repeated values; the same loop works on the real file:

Bash Parse a kernel command line

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Using UUID= instead of /dev/sda2 for the root device matters because device names can change when you add a disk. Every step of this boot also leaves traces: dmesg for the kernel part and journalctl -b for systemd and services. If the machine does not boot at all, the usual fixes are to pick an older kernel in the GRUB menu, add systemd.unit=rescue.target for one boot, or boot a live USB, mount the root and regenerate the initramfs (update-initramfs -u, dracut --force) and GRUB configuration.

Other boot loaders, and the old SysV boot

GRUB is the default, not the only choice. A boot loader has one job: find a kernel (and an initramfs) and start it with the right command line. These are the ones you may meet:

Loader Where you meet it Notes
GRUB 2 Most distributions Scriptable, reads many file systems, config generated into grub.cfg
systemd-boot Arch, Fedora (optional), Pop!_OS UEFI only, tiny, plain-text entries on the EFI System Partition
SYSLINUX / ISOLINUX Installer ISOs, USB sticks, PXE boot Simple config, good for live media and network boot
LILO Old systems (retired 2015) Stored raw disk locations of the kernel, so you had to re-run lilo after every kernel change
LOADLIN DOS-era dual-boot Started Linux from a running DOS
EFISTUB Minimal setups The kernel itself is an EFI program; firmware (or efibootmgr) starts it directly, no loader at all
coreboot and U-Boot Chromebooks, routers, ARM boards Replace the firmware or play the loader on embedded hardware

The earliest stage is the same everywhere. After power-on the firmware (BIOS or UEFI) runs the power-on self-test (POST), finds the boot device, and hands over to the loader: on BIOS from the first 446 bytes of the disk (MBR) and on UEFI from a .efi file in the EFI System Partition. After the kernel unpacks the initramfs (a compressed cpio archive; see cpio), it mounts the real root file system and starts PID 1.

Before systemd the first program was SysVinit. It read /etc/inittab, entered a runlevel (0 halt, 1 single-user, 2 to 5 multi-user variants, 6 reboot) and ran the scripts linked in /etc/rc3.d/ and the other rcN.d folders: names starting with S started, names starting with K stopped, and the number set the order. On such systems runlevel prints the current level, init 6 reboots and service or chkconfig manages services. The final stage of boot also differs: after the services, a display manager (GDM, LightDM, SDDM) starts the graphical login if the system target is graphical.

Useful kernel parameters to pass to any loader include single or systemd.unit=rescue.target (rescue mode), fsck.mode=force (check file systems at boot), pci=noacpi (work around ACPI bugs) and quiet splash (hide boot messages).

Reading a kernel version

uname -r prints a release such as 6.8.0-45-generic. It is a convention, not a standard, but you can split it reliably:

Part Example Meaning
Major.minor 6.8 The upstream kernel series; new minor versions come every 9 to 10 weeks
Patch .0 or .12 Stable bug-fix releases of that series
Build -45 The distribution's own package revision (backported fixes)
Flavour -generic The distribution's build variant (generic, lowlatency, cloud, aws)

uname -a adds the host name, the build date and string (#47-Ubuntu SMP, where SMP means multi-processor support) and the machine architecture (x86_64, aarch64; older 32-bit Intel PCs show i686). Compare versions with sort -V and never with plain string comparison.

Bash Split a kernel release and compare versions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Kernel modules

The kernel is not one block. Drivers, filesystems and network features can be built in or compiled as loadable modules, files ending in .ko (sometimes .ko.zst or .ko.xz) under /lib/modules/$(uname -r)/. A module is loaded into the running kernel when needed and can be removed again, with no reboot.

flowchart LR
    H["New hardware appears<br/>(udev event)"] --> M["modprobe MODULE"]
    U["Admin: modprobe, or a boot-time list<br/>in /etc/modules-load.d/"] --> M
    M --> D["Reads modules.dep<br/>(built by depmod)"]
    D --> Dep["Loads dependencies first"]
    Dep --> K["Loads the module into the kernel"]
    K --> O["Module appears in lsmod /<br/>/proc/modules and /sys/module/"]

modprobe is the smart loader: it resolves dependencies and honours the configuration files.

The tools:

CONSOLE
$ lsmod | head -4                    # loaded modules: name, size, use count, used by
Module                  Size  Used by
nf_conntrack          217088  3 nf_nat,xt_conntrack,xt_MASQUERADE
nf_nat                 61440  2 xt_nat,xt_MASQUERADE
$ modinfo nf_nat                     # author, license, parameters, dependencies
$ sudo modprobe dummy                # load a module and its dependencies
$ sudo modprobe -r dummy             # unload (fails if something still uses it)
$ sudo modprobe -n -v dummy          # dry run: show what would happen
$ sudo depmod -a                     # rebuild the dependency map after installing modules
$ sudo insmod ./my.ko                # load ONE file, no dependency handling (for development)
$ sudo rmmod dummy                   # unload, no dependency handling

Prefer modprobe and modprobe -r over insmod and rmmod: they know about dependencies. lsmod simply formats /proc/modules, and you can read that file without any privilege. This script parses a sample of it and answers a question lsmod makes awkward, "which modules does this one rely on?":

Bash Read /proc/modules like lsmod

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Read the "Used by" column as "who depends on me": xt_nat and xt_MASQUERADE need nf_nat, and nf_nat in turn needs nf_conntrack, so you must unload them in the opposite order. The count and the list are different things: the count also includes users that are not modules (for example an open device or an active firewall rule), which is why xt_conntrack shows a count of 1 with an empty list, and why modprobe -r can fail with "Module is in use" even when the list looks empty.

Making module choices permanent

Hardware drivers load automatically when udev sees the device. To control the rest, use three places:

Where Purpose Example line
/etc/modules-load.d/*.conf load at every boot br_netfilter
/etc/modprobe.d/*.conf options and rules for modprobe options snd_hda_intel power_save=1
/etc/modprobe.d/*.conf forbid loading by name blacklist pcspkr

blacklist only stops the automatic alias-based loading; a module another module needs, or one you modprobe explicitly, still loads. For a hard ban use install usb_storage /bin/false, which replaces the load command with something that fails. Changes to modules needed during early boot also require rebuilding the initramfs. A small parser shows the effective result of a set of such lines:

Bash Interpret modprobe.d rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Modules can also take parameters live. Their current values are files: cat /sys/module/NAME/parameters/PARAM shows them, and many can be changed by writing to that file as root. Third-party drivers (graphics cards, VirtualBox, ZFS) are often built with DKMS, which recompiles them for each new kernel. With Secure Boot enabled, such modules must be signed, or the kernel refuses to load them.

sysctl: tuning the running kernel

Many kernel behaviours are adjustable at run time without rebooting. They appear as files under /proc/sys/, and sysctl is the tool that reads and writes them using dotted names that map to the path: net.ipv4.ip_forward is the file /proc/sys/net/ipv4/ip_forward.

CONSOLE
$ sysctl net.ipv4.ip_forward                     # read one value
net.ipv4.ip_forward = 0
$ cat /proc/sys/net/ipv4/ip_forward              # exactly the same value
0
$ sudo sysctl -w net.ipv4.ip_forward=1           # change it now (lost at reboot)
$ sysctl -a | grep swappiness                    # search all tunables
$ sudo sysctl --system                           # re-read every configuration file

sysctl -w is temporary. To keep a setting, put it in a file in /etc/sysctl.d/, named with a number prefix so the order is clear, and apply it with sudo sysctl --system. Files are read in alphabetical order and later values override earlier ones, which is why a file called 99-local.conf is the right place for your own tweaks. Lines are key = value; # and ; start comments.

Tunables worth knowing, grouped by purpose:

Area Key Typical use
Network net.ipv4.ip_forward 1 on routers, VPN gateways and container hosts, otherwise 0
Network net.ipv4.tcp_syncookies 1: resist SYN floods
Network net.core.somaxconn length of the listen queue for busy servers
Memory vm.swappiness 10 to 60: how eagerly to swap out memory (lower on database servers)
Memory vm.overcommit_memory policy for allocating more memory than exists
Files fs.file-max, fs.inotify.max_user_watches system-wide open files; watchers for editors and file syncing
Security kernel.randomize_va_space 2: full address space randomisation
Security kernel.dmesg_restrict, kernel.kptr_restrict hide kernel messages and addresses from ordinary users
Debug kernel.perf_event_paranoid who may use perf (see strace, ltrace, lsof, perf and eBPF Intro)

The script below reads real tunables from /proc/sys and resolves a sample configuration, including the "later file wins" rule. It also tries to change a value, which the unprivileged read-only sandbox refuses, exactly as a normal user would be refused on a real machine (use sudo there):

Bash sysctl names, files and precedence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last section shows why numeric prefixes matter: Docker's file turns forwarding on, your own 99-local.conf lowers swappiness, and nothing you do in 10-defaults.conf can undo either because it is read first. When a setting "does not stick", look for a later file that overrides it: sudo sysctl --system prints every file it applies, in order. Some tunables only exist when the module that owns them is loaded: net.bridge.* needs br_netfilter, which links this lesson's two topics together.

Finally, tune with evidence. Copying a list of "performance sysctls" from a blog can reduce stability; change one setting at a time, measure the effect, and record why in a comment next to the line.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the job of the initramfs?

    The kernel may need a driver (RAID, LVM, encryption, a storage controller) that lives on the disk it cannot yet read. The initramfs is loaded into memory by the bootloader and contains those pieces.
  2. 2. You changed GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub. What else is needed?

    grub.cfg is generated from /etc/default/grub and /etc/grub.d. The change only takes effect after update-grub (Debian family) or grub2-mkconfig -o (RHEL family).
  3. 3. Why prefer modprobe over insmod?

    insmod loads exactly one .ko file. modprobe reads the dependency map from depmod, loads required modules first and applies options, blacklists and install rules.
  4. 4. You run sudo sysctl -w vm.swappiness=10. What happens after a reboot?

    sysctl -w changes the running kernel only. Persist it in a file such as /etc/sysctl.d/99-local.conf and apply with sysctl --system.
  5. 5. Two files in /etc/sysctl.d set the same key: 10-base.conf has vm.swappiness = 60 and 99-local.conf has vm.swappiness = 10. Which value is active after sysctl --system?

    Files are applied in alphabetical order, so later settings override earlier ones. That is why local overrides get a high number.

Summary

  • Boot chain: firmware (UEFI/BIOS) -> GRUB -> kernel + initramfs -> systemd (PID 1) -> target. dmesg and journalctl -b show what happened; systemd-analyze blame finds slow units.
  • Edit /etc/default/grub and regenerate with update-grub or grub2-mkconfig; never edit grub.cfg. Press e in the menu for a one-time change to the kernel command line, visible later in /proc/cmdline.
  • Useful parameters: root=UUID=, ro, quiet, console=, systemd.unit=rescue.target, nomodeset, init=/bin/bash for recovery.
  • Modules are .ko files in /lib/modules/$(uname -r)/. Use lsmod, modinfo, modprobe (with dependencies) and modprobe -r; persist choices in /etc/modules-load.d/ and /etc/modprobe.d/ (options, blacklist, install ... /bin/false).
  • sysctl maps dotted names to /proc/sys/...; -w is temporary, files in /etc/sysctl.d/ persist and later files win; sysctl --system applies them.
  • Change one tunable at a time, measure, and document the reason.

Return to the overview: Linux cheat sheet