Skip to content
Tutorials Center
My learning

Topics

Lesson 52 of 95 ยท Processes and the System

Packages, Services and systemd

How Linux installs software and runs services: apt, dnf and pacman, dpkg, systemctl, unit files, journalctl, timers, cron syntax, and the history of init systems.

On Linux you rarely download a program from a web page and run an installer. Software comes in packages managed by your distribution, and long-running programs such as web servers, databases and schedulers are started and supervised by a service manager. Knowing both layers is the daily work of every Linux administrator, and it is also what you need to ship your own scripts to a server and keep them running.

This lesson explains what a package manager does and how apt, dnf and pacman differ, what systemd is, how to control services with systemctl, how to read their logs with journalctl, what a unit file and a timer look like, and how cron expressions work. Our sandbox has no systemd, no crontab and no root, so most of the system commands below are read-only; the runnable examples parse the real file formats and check the cron syntax with a script you can reuse. It builds on Users, Groups and sudo and Processes.

What a package manager does

A package is an archive with the files of a program (binaries, libraries, configuration, documentation) plus metadata: name, version, a description, and the list of other packages it depends on. A package manager is a tool that:

  1. downloads packages from repositories (servers run by the distribution) and checks their cryptographic signatures;
  2. works out and installs all the dependencies needed;
  3. records which file belongs to which package, so that it can update or remove the software cleanly;
  4. upgrades everything together, which is how security fixes reach your system.
TEXT
   repository (server)          your machine
   +------------------+  index   +-----------------------------+
   | nginx_1.22.deb   | -------> | apt update  (refresh lists) |
   | libssl3.deb      | download | apt install nginx           |
   | ...              | -------> |   -> also libssl3 (dependency)
   +------------------+          |   -> files copied, recorded in a database
                                 +-----------------------------+

There are two layers on most systems. A low-level tool installs a single package file and does not fetch anything (dpkg for Debian, rpm for Red Hat). A high-level tool talks to the repositories and resolves dependencies (apt, dnf, pacman, zypper).

The families

Family Distributions Package file Low-level High-level
Debian Debian, Ubuntu, Mint .deb dpkg apt, apt-get
Red Hat Fedora, RHEL, Rocky, Alma .rpm rpm dnf (older: yum)
Arch Arch, Manjaro .pkg.tar.zst pacman
SUSE openSUSE .rpm rpm zypper
Alpine Alpine (containers) .apk apk

Installing software changes the system, so these commands need root, and there is no network in our sandbox. These are read-only cheat-sheet examples; replace sudo with nothing when you are root:

CONSOLE
# Debian / Ubuntu
$ sudo apt update                    # refresh the package lists
$ sudo apt install nginx             # install (and its dependencies)
$ apt search "web server"            # search names and descriptions
$ apt show nginx                     # details, version, dependencies
$ sudo apt upgrade                   # upgrade installed packages
$ sudo apt remove nginx              # remove, keep configuration
$ sudo apt purge nginx               # remove including configuration
$ sudo apt autoremove                # remove no-longer-needed dependencies

# Fedora / RHEL / Rocky
$ sudo dnf install nginx
$ sudo dnf upgrade
$ sudo dnf remove nginx
$ dnf search "web server"

# Arch
$ sudo pacman -Syu                   # refresh and upgrade everything
$ sudo pacman -S nginx               # install
$ sudo pacman -Rns nginx             # remove with unneeded dependencies
$ pacman -Ss "web server"            # search

On Debian and Ubuntu, run apt update before apt install; otherwise the local package lists may be stale and the package or its newest version can be "not found". In scripts, prefer apt-get with -y and DEBIAN_FRONTEND=noninteractive, because apt warns that its interface is not stable for scripting.

Asking questions about installed packages

The low-level tools tell you what a package contains and where a given file came from, which is invaluable when a command or file is "just there" and you wonder who owns it:

CONSOLE
$ dpkg -l | grep nginx               # list installed packages (Debian)
$ dpkg -L nginx                      # every file the package installed
$ dpkg -S /usr/sbin/nginx            # which package owns this file?
$ rpm -qf /usr/sbin/nginx            # the same on Red Hat systems
$ apt-cache policy nginx             # installed version and candidate versions

The tools themselves are present in our sandbox, but the machine has no package database, so they have nothing to report. That is a good example of a failure you may meet in broken containers:

Bash dpkg without a package database (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The first message says dpkg-query: error: failed to open package info file '/var/lib/dpkg/status' for reading: that file is the package database, and this sandbox image was trimmed down without it. The second command stops at once with E: Could not open lock file /var/lib/dpkg/lock-frontend and are you root?: installing needs root (and the database folder, which is missing here). On a real server you would run it with sudo and it would work.

Good package habits

  • Install from the distribution first. You get security updates automatically. Download a .deb or .rpm by hand only from vendors you trust.
  • Never curl ... | sudo bash from sites you do not control; read the script first.
  • Keep the system updated, and use unattended security updates on servers.
  • Pin versions in containers and CI (apt-get install nginx=1.22.1-9) so builds are repeatable.
  • Know your language's package managers too. pip, npm, cargo, gem and composer are separate worlds; do not mix them with system packages in system directories.
  • Containers (Docker) and universal formats (Snap, Flatpak, AppImage) bundle dependencies with the application, trading disk space for independence from the distribution.

From init scripts to systemd

When the kernel has finished starting, it launches exactly one user-space program, PID 1, which starts everything else and adopts orphaned processes (see kill, Signals and Process Control). That program is the init system.

Era Init system How it works
1980s to 2000s SysV init Shell scripts in /etc/init.d/, started one after another by runlevel; service nginx start
2006 Upstart (Ubuntu) Event-driven, replaced SysV scripts for a while
2010 onward systemd Declarative unit files, starts services in parallel, supervises and restarts them, handles logging, timers, sockets, mounts
Alternatives OpenRC, runit, s6, dinit Simpler designs (Alpine, Void, Gentoo); containers often use none, or a tiny init

Today nearly every mainstream distribution uses systemd. It is both loved (fast boot, consistent tools, dependency handling) and criticized (large scope), but you will meet it everywhere, so it is worth learning well.

systemd units

A unit is a thing systemd manages, described by a small text file. The type is the file extension:

Unit type Describes
.service A program to run (a web server, a script)
.timer A schedule that triggers another unit (a modern cron)
.socket A listening socket that starts a service on demand
.target A group of units, like old runlevels (multi-user.target)
.mount, .path, .slice Mount points, file watching, resource groups

System unit files from packages live in /usr/lib/systemd/system/ (or /lib/systemd/system/); your own go to /etc/systemd/system/, which takes precedence. A minimal service for a script of your own looks like this (read-only example):

INI
# /etc/systemd/system/report.service
[Unit]
Description=Nightly sales report
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=reports
WorkingDirectory=/srv/reports
ExecStart=/usr/local/bin/make-report.sh --quiet
Environment=REPORT_DIR=/srv/reports/out

[Install]
WantedBy=multi-user.target
  • [Unit] describes the service and its ordering: After= means "start later than", Wants=/Requires= pull other units in.
  • [Service] says how to run it. Type=simple (default) is a program that stays in the foreground, Type=oneshot is a program that runs and ends, Type=forking is an old-style daemon. Restart=on-failure supervises it; User= gives it an unprivileged identity (least privilege).
  • [Install] says when to start it at boot (WantedBy=), used by systemctl enable.

The file format is INI-like, so even without systemd you can read it with tools you know. This script lists the services in a directory of unit files with their ExecStart commands:

Bash Reading unit files with awk

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The awk script remembers the current section in a variable and reacts only to keys in [Service]. Real unit files also allow comments, line continuations and multiple ExecStart lines for oneshot, so use systemctl cat NAME and systemd-analyze verify FILE on a real system to check them.

systemctl: controlling services

systemctl is the main command. All of these need a running systemd, so they are read-only here:

CONSOLE
$ systemctl status nginx             # state, PID, last log lines
$ sudo systemctl start nginx         # start now
$ sudo systemctl stop nginx          # stop now
$ sudo systemctl restart nginx       # stop and start
$ sudo systemctl reload nginx        # re-read config without dropping connections
$ sudo systemctl enable nginx        # start automatically at boot
$ sudo systemctl enable --now nginx  # enable and start in one step
$ sudo systemctl disable nginx
$ systemctl is-active nginx          # prints "active"; exit code 0 if so
$ systemctl is-enabled nginx
$ systemctl list-units --type=service --state=running
$ systemctl list-timers              # scheduled timer units
$ sudo systemctl daemon-reload       # after editing unit files
$ systemctl cat nginx                # show the unit file(s) in effect
$ systemctl --user status myapp      # per-user services, no root needed

Two details beginners trip over: start and enable are independent (start now, start at boot), and after you edit a unit file you must run systemctl daemon-reload before the change is noticed. Because is-active and is-enabled set the exit status, they work directly in scripts: if systemctl is-active --quiet nginx; then ....

Our sandbox shows what happens without systemd:

Bash No systemd here

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Inside many Docker containers you see the same: PID 1 is the application or a tiny init program and there is no systemctl. Scripts that must work in both worlds can test for /run/systemd/system, the directory that exists only when systemd is the running init system. Containers use the container runtime as the service manager instead.

Masking, and what start, stop and restart really mean

disable only removes the boot-time link, so another unit that lists the service in Wants= or Requires= can still start it. Masking goes further: sudo systemctl mask nginx links the unit file to /dev/null, so nothing (not you, not a dependency, not systemctl start) can start it. sudo systemctl unmask nginx undoes it. Use it for a service that must never run, such as a second firewall frontend that would fight with the one you chose. systemctl list-unit-files --state=masked lists every masked unit.

Some vocabulary used in all these commands. A daemon is a program that runs in the background without a terminal, usually waiting for work (sshd, nginx, cron); its name often ends with d. A service is how systemd manages a daemon: a unit that says how to start it, stop it, restart it on failure and what it depends on. stop ends the process, restart stops and starts it (briefly dropping connections), reload asks the running process to re-read its configuration (a SIGHUP or a special command) and try-restart restarts only if it is already running.

Unit files can also sandbox a service. A few directives worth knowing: PrivateTmp=true gives the service its own empty /tmp; ProtectSystem=full (or strict) mounts /usr, /boot and /etc read-only for it; ProtectHome=true hides /home; ReadOnlyPaths= and ReadWritePaths= make exceptions; NoNewPrivileges=true stops it from gaining rights. systemd-analyze security nginx scores a unit.

To control services on another machine, use systemctl -H user@host status nginx, which runs over SSH, or systemctl --host. If a user without root must manage one unit, grant it through polkit with a rule file under /etc/polkit-1/rules.d/ that allows org.freedesktop.systemd1.manage-units for a group, instead of giving a full sudo right.

apt-get, dpkg and dnf in more detail

The earlier commands cover daily work. Scripts and troubleshooting need a few more options.

CONSOLE
$ sudo apt-get install -y --no-install-recommends nginx   # skip "recommended" extras (small images)
$ sudo apt-get install --install-suggests pkg             # also pull in "suggested" packages
$ sudo apt-get -s install nginx            # -s / --simulate: show what WOULD happen
$ sudo apt-get -d install nginx            # -d: download only, do not install
$ sudo apt-get -f install                  # -f: fix broken dependencies
$ sudo apt-get --reinstall install nginx   # reinstall the same version
$ sudo apt-get install --only-upgrade nginx   # upgrade only if already installed
$ sudo apt-get -y --assume-no upgrade      # --assume-no answers every question with no
$ sudo apt-get dist-upgrade                # upgrade and allow adding or removing packages
$ sudo apt-get autoremove --purge          # remove unused dependencies and their config
$ sudo apt-get clean                       # empty the downloaded package cache
$ sudo apt-get -V upgrade                  # -V: show old and new version numbers

apt-get has a stable output meant for scripts; apt is the friendlier interactive front end with a progress bar. -y answers "yes" to prompts: use it in automation, never as a reflex.

dpkg is the layer beneath. It installs a single .deb file and checks nothing in repositories:

CONSOLE
$ sudo dpkg -i package.deb      # install (if a dependency is missing, run: sudo apt-get -f install)
$ dpkg -l | grep nginx          # list packages and their state ("ii" = installed)
$ dpkg -L nginx                 # files a package installed
$ dpkg -S /usr/sbin/nginx       # which package owns this file?
$ sudo dpkg -r nginx            # remove, keep configuration files
$ sudo dpkg -P nginx            # purge: remove and delete configuration
$ sudo dpkg --audit             # list packages that are only partly installed
$ sudo dpkg --configure -a      # finish interrupted installs

On Red Hat systems dnf records every change as a numbered transaction, which makes mistakes reversible:

CONSOLE
$ sudo dnf upgrade                   # upgrade everything (or name one package)
$ sudo dnf reinstall nginx           # reinstall a package
$ dnf history                        # numbered list of past transactions
$ dnf history info 12                # what transaction 12 changed
$ sudo dnf history undo 12           # reverse it
$ sudo dnf system-upgrade download --releasever=41   # plugin for a release upgrade

journalctl: reading the logs

systemd collects the standard output and error of every service, plus system messages, in a binary journal, queried with journalctl. Classic text logs in /var/log/ still exist (see Disks, Mounts, Logs and Monitoring).

CONSOLE
$ journalctl -u nginx                    # all messages of one unit
$ journalctl -u nginx -f                 # follow, like tail -f
$ journalctl -u nginx --since "1 hour ago"
$ journalctl -u nginx -n 50 --no-pager   # last 50 lines
$ journalctl -p err -b                   # errors and worse since this boot
$ journalctl -k                          # kernel messages (like dmesg)
$ journalctl --disk-usage                # how much space the journal uses

Because the journal captures what a service prints, your own scripts need no log-file code when run as a service: write to stdout and stderr, and read it back with journalctl -u. Use logger -t myscript "message" for a message from any script, which appears in the journal and syslog.

Targets, boot analysis and the other *ctl tools

Targets replace runlevels

Old SysV init had numbered runlevels. systemd replaced them with targets: unit files (something.target) that do nothing themselves but group other units and mark a state the system should reach. Targets are how boot is ordered: a target Requires= or Wants= its members and uses After= to wait for earlier targets.

Runlevel Target Meaning
0 poweroff.target Shut down
1 rescue.target Single-user rescue shell
3 multi-user.target Full system, text login, services
5 graphical.target multi-user plus the desktop
6 reboot.target Reboot

systemctl get-default shows which target boots by default, systemctl set-default multi-user.target changes it, and systemctl isolate rescue.target switches the running system to another one (stopping everything not part of it). The [Install] line WantedBy=multi-user.target from earlier is the link: systemctl enable creates a symlink inside multi-user.target.wants/, so starting that target starts your service. systemctl list-dependencies multi-user.target prints the whole tree.

Remember that WantedBy= says whether a unit starts with a target, while After= and Before= say in which order. The two are independent, and a unit with no ordering can start in parallel with everything else, which is why systemd boots quickly. The next example sorts a few made-up unit files by their After= lines, the same graph systemd walks:

Bash Order units by their After= lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

tsort (topological sort) prints the units so that every one comes after the units it waits for. When two units have no dependency between them, either order is valid, and systemd starts them at the same time. A loop such as a after b after a is impossible; systemd breaks it by dropping a job and reports "ordering cycle" in the journal.

systemd-analyze: why is boot slow?

systemd-analyze measures the boot:

  • systemd-analyze prints total time split into firmware, loader, kernel and userspace.
  • systemd-analyze blame lists units by the time they took to start, slowest first. A service at the top is the first suspect, but note that units start in parallel, so the sum is bigger than the boot time.
  • systemd-analyze critical-chain shows the chain of units that actually decided the boot time: each line is a unit that had to finish before the next one could start. This is the list worth optimizing.
  • systemd-analyze verify app.service checks a unit file for mistakes before you install it, and systemd-analyze security app.service scores how well the service is sandboxed (see System Hardening).

The other *ctl tools

systemd ships a family of small control commands, all with similar syntax:

Command Purpose Example
hostnamectl Show or set the host name and machine info hostnamectl set-hostname web01
timedatectl Time zone, clock and NTP sync state timedatectl set-timezone Europe/Berlin, timedatectl status
localectl System locale and keyboard layout localectl set-locale LANG=en_US.UTF-8
loginctl Login sessions and seats loginctl list-sessions, loginctl terminate-user alice
systemd-cgls The cgroup tree: which service owns which processes systemd-cgls
systemd-cgtop Live CPU, memory and I/O per cgroup (like top for services) systemd-cgtop
journalctl Query the journal journalctl -b -p err
systemctl poweroff, reboot, suspend Power control (older shutdown -h now, reboot, halt) systemctl reboot

Because systemd places every service in its own cgroup, systemd-cgls is the cleanest way to see which processes belong to which service, including children that a plain ps tree would scatter. A stopped service takes its whole cgroup with it, so no stray child processes survive. None of these commands work in the sandbox (there is no systemd as PID 1), so treat them as read-only reference and try them on a real machine or a VM.

Timers and cron: running things on a schedule

Two systems run commands on a schedule. cron is the classic, universal one. systemd timers are the modern alternative with logging and dependency support. A timer is a .timer unit paired with a .service of the same name (read-only example):

INI
# /etc/systemd/system/report.timer
[Unit]
Description=Run the report every weekday at 02:30

[Timer]
OnCalendar=Mon..Fri 02:30
Persistent=true

[Install]
WantedBy=timers.target

Persistent=true runs a missed job after the machine was off, which cron cannot do. Enable it with systemctl enable --now report.timer and check with systemctl list-timers.

cron syntax

Each user has a crontab (edit with crontab -e, list with crontab -l); the system has /etc/crontab and the folders /etc/cron.d/, /etc/cron.daily/ and friends. A schedule line has five time fields and then the command:

TEXT
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ minute        (0-59)
โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€ hour          (0-23)
โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€ day of month  (1-31)
โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€ month         (1-12)
โ”‚ โ”‚ โ”‚ โ”‚ โ”Œโ”€ day of week   (0-6, Sunday = 0; 7 also means Sunday on most systems)
โ”‚ โ”‚ โ”‚ โ”‚ โ”‚
* * * * *  command to run

Each field accepts:

Syntax Meaning Example
* every value * * * * * = every minute
5 exactly this value minute 5
1,15 a list days 1,15
9-17 a range hours 9-17
*/10 every 10th value minutes */10
10-50/20 a range with a step minutes 10, 30, 50

More examples: 0 3 * * * is daily at 03:00; 30 2 * * 1-5 is 02:30 on weekdays; 0 0 1 * * is midnight on the first of each month; */15 9-17 * * 1-5 is every 15 minutes during office hours. Shortcuts exist too: @reboot, @hourly, @daily, @weekly, @monthly.

A few facts that cause most cron bugs:

  • If both day-of-month and day-of-week are restricted (not *), the job runs when either matches, not both.
  • The cron environment is minimal (a short PATH, no aliases, SHELL=/bin/sh): use absolute paths or set PATH= at the top of the crontab (see Environment, Profiles and Configuration).
  • The percent sign % has a special meaning in a crontab command (newline); escape it as \%, e.g. in date +\%F.
  • Cron mails any output to the owner. Redirect it: >> /var/log/job.log 2>&1.
  • Times use the server's time zone; servers should run UTC.
  • Use flock -n /tmp/job.lock command so that a slow run cannot overlap with the next one.

Checking a cron expression with a script

Nobody wants to wait a whole day to find out whether a schedule is right. Here is a small matcher written in Bash. It splits an expression into the five fields and tests whether a given moment matches, then lists the next runs after a fixed starting time. printf '%(...)T' is a Bash builtin that formats an epoch time without starting date, and TZ=UTC keeps the result independent of the machine.

Bash A cron expression checker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Read the first schedule: every twenty minutes, in hours 9 and 10, on Monday to Friday. Starting on Friday 08:50, the first run is 09:00, then 09:20, 09:40 and 10:00. The weekly schedule 0 3 * * 0 finds the coming Sundays at 03:00, and 30 2 1,15 * * shows the first and fifteenth of the month. The function day_matches implements the "either may match" rule for the day fields, which is the rule people forget, and next_runs jumps over whole days and hours that cannot match so the search stays fast (the weekly schedule would otherwise test 10,000 separate minutes). For production use crontab -l, the systemd-analyze calendar command for timers, or an online cron explainer, but the exercise shows how little magic there is in the notation. See Functions and Arithmetic and Numbers for the Bash features used.

Validating before installing

A syntactically wrong line in a crontab is silently ignored by some cron versions, and crontab -e only complains about the worst errors. A cheap first line of defense is to count the fields:

Bash Counting cron fields

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last line is reported as bad because the @daily shortcut replaces the five fields with a single word; a real validator has to know both forms. The point is to check something automatically before a typo stays hidden until the night the job should have run.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main advantage of a package manager like apt or dnf over downloading and unpacking software by hand?

    A package manager tracks which files belong to which package, installs dependencies automatically, verifies signatures and keeps everything upgradable. It needs root and, to install new software, network access to the repositories.
  2. 2. You edited /etc/systemd/system/app.service while the service was running. What do you have to do for systemd to notice the change?

    systemd caches unit files. After editing one, run systemctl daemon-reload so it re-reads them, and restart the service to apply the new settings.
  3. 3. What is the difference between systemctl start nginx and systemctl enable nginx?

    start and enable are independent. enable only creates the links that start the unit at boot; enable --now does both.
  4. 4. The cron line 30 2 1,15 * 1 is installed. When does it run?

    When both day-of-month and day-of-week are restricted, cron runs the job if either field matches, not both. It runs on the 1st, the 15th and every Monday.
  5. 5. A script runs perfectly in your terminal but fails when started by cron with command not found. Which fix is most likely to work?

    cron starts jobs with a very small environment, usually PATH=/usr/bin:/bin and no aliases. Full paths or an explicit PATH setting make the job independent of your interactive configuration.

Summary

  • A package manager downloads signed packages from repositories, resolves dependencies and records installed files: apt/dpkg on Debian and Ubuntu, dnf/rpm on Fedora and RHEL, pacman on Arch. Run apt update before apt install.
  • PID 1 is the init system. SysV init scripts gave way to systemd, which describes services in declarative unit files ([Unit], [Service], [Install]).
  • systemctl starts, stops, enables, restarts and inspects units; start and enable are separate; run daemon-reload after editing a unit.
  • journalctl -u NAME reads a service's logs; services only need to write to stdout and stderr.
  • systemd timers (OnCalendar=, Persistent=) and cron both schedule jobs. A cron line has five time fields (minute, hour, day of month, month, day of week) followed by the command.
  • Remember cron's quirks: the minimal environment, % needing an escape, the "either day field" rule, and overlapping runs (use flock).
  • Targets (multi-user.target, graphical.target) replaced runlevels; systemd-analyze blame and critical-chain explain slow boots; hostnamectl, timedatectl, localectl and systemd-cgls are the companion tools.
  • Our sandbox has no systemd, no crontab, no package database and no root, so commands that need them are read-only; the formats can still be parsed and checked with ordinary shell tools.

Next: Networking Basics from the Shell