Skip to content
Tutorials Center
My learning

Topics

Lesson 87 of 95 ยท System Administration

Logging: journald, rsyslog and logrotate

Read, filter and rotate Linux logs: how journald and rsyslog collect messages, journalctl filters by unit, time and priority, and logrotate keeps disks from filling.

Every service on a Linux machine talks about what it is doing, and almost all of that talk ends up in a log. When something breaks at 3 a.m., the log is the first place you look, so you need to know where the messages are stored, how to filter a mountain of them down to the five lines that matter, and how to stop them from filling the disk.

On a modern distribution two systems cooperate. systemd-journald collects messages from every service into a binary journal, and rsyslog (or a similar daemon) can write the same messages into classic text files under /var/log. This lesson explains both, shows the journalctl filters you will use every day, and ends with logrotate. The sandbox has no journal and no root, so the real commands are read-only; the runnable examples work on sample log text with the same shapes, using the tools from Disks, Mounts, Logs and Monitoring and Packages, Services and systemd.

How a log message travels

A program can write to its own file, to standard output, or hand the message to the system logger. Under systemd, a service's output is captured automatically, so a plain echo in a script run as a service already lands in the journal.

flowchart LR
  A["Service stdout/stderr"] --> J["journald"]
  B["syslog() calls, logger"] --> J
  K["Kernel (dmesg)"] --> J
  J --> S["Binary journal<br/>/var/log/journal"]
  J --> R["rsyslog"]
  R --> F["Text files<br/>/var/log/syslog, auth.log"]
  F --> L["logrotate (weekly)"]

Messages flow into journald, which stores them in the journal and can forward them to rsyslog for classic text files.

Every message carries a priority (severity), from 0 to 7. These are the syslog levels, and journalctl -p understands both numbers and names:

Number Name Meaning
0 emerg System is unusable
1 alert Act immediately
2 crit Critical condition
3 err Error
4 warning Warning
5 notice Normal but significant
6 info Informational
7 debug Debug detail

Messages also have a facility (auth, cron, daemon, kern, mail, ...) that says which part of the system sent them. Rsyslog uses facility plus priority to decide which file a message goes to.

Reading the journal with journalctl

journalctl queries the journal. Run as a normal user you only see your own messages unless you are in the systemd-journal or adm group; sudo shows everything.

CONSOLE
$ journalctl -u nginx.service            # one service
$ journalctl -u nginx -f                 # follow, like tail -f
$ journalctl -n 50                       # last 50 lines
$ journalctl -b                          # this boot only
$ journalctl -b -1                       # the previous boot
$ journalctl --since "2 hours ago"       # relative time
$ journalctl --since "2026-03-01 08:00" --until "2026-03-01 09:30"
$ journalctl -p err                      # err and worse (0-3)
$ journalctl -p warning..err             # a range of priorities
$ journalctl -k                          # kernel messages only
$ journalctl -u nginx -o json-pretty     # structured output
$ journalctl --disk-usage                # how much space the journal uses

Read-only: the sandbox has no journalctl. Filters combine with AND, so journalctl -u nginx -p err --since today means "nginx, errors or worse, since midnight". Each journal entry also has fields such as _SYSTEMD_UNIT, _PID and PRIORITY, and you can match on any of them: journalctl _PID=812.

The journal is stored in a binary format, so grep cannot search the files directly. You pipe journalctl output (plain text) into your usual tools, and the shape of that text is predictable. Here it is on sample data in the default "short" format:

Bash Filtering journal-style lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The same pattern, "filter, extract a field, count", works on live journalctl output. Prefer the built-in filters (-u, -p, --since) first, because they are faster and cannot be fooled by text inside a message.

Classic log files and rsyslog

Many distributions (Debian and Ubuntu especially) also run rsyslog, which writes plain text files. The typical ones:

File Content
/var/log/syslog (Debian) or /var/log/messages (RHEL) General system messages
/var/log/auth.log or /var/log/secure Logins, sudo, SSH
/var/log/kern.log Kernel messages
/var/log/dpkg.log, /var/log/apt/ Package installs
/var/log/nginx/access.log, error.log Per-application logs

Rsyslog is configured in /etc/rsyslog.conf and /etc/rsyslog.d/*.conf. The core rule syntax is facility.priority destination:

TEXT
# send all auth messages to their own file
auth,authpriv.*              /var/log/auth.log

# everything except auth and mail goes to syslog; "-" means do not sync every line
*.*;auth,authpriv.none       -/var/log/syslog

# mail errors and worse
mail.err                     /var/log/mail.err

# forward everything to a central log server over TCP (@@ = TCP, @ = UDP)
*.*                          @@loghost.example.com:514

Forwarding to a central server is how fleets avoid logging into each machine. Scripts can write into the same system with logger:

CONSOLE
$ logger -t backup -p user.notice "nightly backup finished"
$ logger -t backup -p user.err "backup failed: disk full"
$ journalctl -t backup                 # find them again by tag

The tag (-t) becomes the program name in the output, so choose one per script.

By default the journal may live only in memory (/run/log/journal) and vanish at reboot. Creating the directory /var/log/journal, or setting Storage=persistent in /etc/systemd/journald.conf, makes it survive reboots. The same file limits its size with SystemMaxUse= (for example 500M).

Keeping logs under control with logrotate

A log that is only ever appended to will eventually fill the disk. logrotate runs once a day (from a systemd timer or /etc/cron.daily) and, for each log named in its configuration, renames the current file, starts a fresh one, compresses old copies and deletes the oldest. The result is a series like app.log, app.log.1, app.log.2.gz, app.log.3.gz.

Per-application rules live in /etc/logrotate.d/. A complete example for a hypothetical service:

TEXT
/var/log/myapp/*.log {
    weekly                 # rotate once a week (daily, monthly also work)
    rotate 8               # keep 8 old copies, delete the rest
    compress               # gzip old copies ...
    delaycompress          # ... but leave the newest old one uncompressed
    missingok              # no error if the log is absent
    notifempty             # skip empty files
    create 0640 myapp adm  # mode, owner, group of the new file
    sharedscripts
    postrotate
        systemctl kill -s HUP myapp.service   # tell the program to reopen its log
    endscript
}

The important question is how the program learns about the new file. If it keeps the old file open, it will go on writing into app.log.1. The two answers are a postrotate signal (usually HUP) so it reopens its log, or copytruncate, which copies the content away and empties the original in place (simpler, but a few lines written during the copy can be lost).

Useful commands (read-only here, as they need root):

CONSOLE
$ sudo logrotate -d /etc/logrotate.d/myapp   # debug: show what would happen, change nothing
$ sudo logrotate -f /etc/logrotate.conf      # force a rotation now
$ cat /var/lib/logrotate/status              # when each log was last rotated

The rotation itself is only renames, so you can model it in a few lines of shell:

Bash A tiny rotation, by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Only three old copies survive, however many days pass: that is rotate 3. Real logrotate does the same bookkeeping and adds age checks, size limits (size 100M), date-based names (dateext) and scripts.

Finding problems fast

A short routine that works on almost any incident:

  1. journalctl -p err -b for errors since boot.
  2. journalctl -u SERVICE --since "30 min ago" for the failing service.
  3. grep -i 'out of memory\|oom' /var/log/syslog (or journalctl -k | grep -i oom) for the kernel killing processes.
  4. df -h /var and journalctl --disk-usage to be sure logs are not what filled the disk.
  5. tail -f the application's own log while you reproduce the problem.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which command shows only error-level and worse messages from the nginx service since the last boot?

    -u selects the unit, -p err keeps priorities 0 to 3 and -b limits the output to the current boot. The journal is binary, so grep on its files does not work, and -k means kernel messages.
  2. 2. Why does a service sometimes keep writing into app.log.1 after logrotate ran?

    Renaming a file does not change which file an open descriptor points to. Use a postrotate signal such as HUP so the program reopens its log, or copytruncate.
  3. 3. What does rotate 8 with weekly mean in a logrotate rule?

    weekly sets how often, rotate 8 how many old copies are kept before the oldest is deleted, roughly two months of history.
  4. 4. What is the job of the tag in logger -t backup "done"?

    The tag identifies the sender of the message. Priority is set with -p, and rsyslog rules route messages to files.

Summary

  • journald collects output from services, the kernel and logger into a binary journal; rsyslog can also write classic text files in /var/log.
  • Priorities run from 0 (emerg) to 7 (debug); journalctl -p err means errors and worse.
  • Useful journalctl filters: -u (unit), -b (boot), --since/--until, -p, -k, -f and -o json-pretty. They combine with AND.
  • Pipe journal output into grep, awk and sort | uniq -c for counting, as with any text.
  • Rsyslog rules look like facility.priority destination; @@host forwards over TCP.
  • logrotate renames, compresses and deletes old logs; programs must reopen their log (postrotate signal) or you use copytruncate.
  • Make the journal persistent and cap its size in /etc/systemd/journald.conf.

Next: Namespaces, cgroups and chroot