Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 37 ยท Monitoring and Logs

System Logs with journald and rsyslog

Read and manage Linux logs: journalctl filters and flags, persistent journals, syslog facilities and priorities, /var/log files, rsyslog rules, and forwarding logs to a central server.

Logs are the first place to look when something breaks, and the only evidence you have after a security incident. On modern Linux, two systems cooperate: systemd-journald collects messages from the kernel, services and stdout of units into a binary journal, and rsyslog (on many distributions) writes classic text files under /var/log and can forward them to other hosts. This lesson teaches you to read the journal fluently, understand the syslog model, and set up basic forwarding.

The site cannot show your machine's logs, so the commands are read-only and the log excerpts are examples. The Python stand-ins parse realistic lines so you can practise the filtering logic.

Reading the journal with journalctl

Terminal
journalctl                       # everything, oldest first (pager)
journalctl -e                    # jump to the end
journalctl -f                    # follow new entries, like tail -f
journalctl -n 50                 # last 50 lines
journalctl -b                    # current boot
journalctl -b -1                 # previous boot
journalctl --list-boots
journalctl -u nginx              # one unit
journalctl -u nginx -u php8.3-fpm
journalctl -p err                # priority err and worse
journalctl -p warning..err
journalctl --since "2025-11-12 08:00" --until "09:30"
journalctl --since "1 hour ago"
journalctl -k                    # kernel messages (like dmesg)
journalctl _PID=812
journalctl _UID=1001
journalctl /usr/sbin/sshd        # by executable
journalctl -o json-pretty -n 1   # structured fields
journalctl -g 'failed|denied'    # grep inside messages
journalctl --disk-usage

Combine filters: journalctl -u ssh -p warning --since yesterday shows only warnings from the SSH service in the last day. The journal is structured: every entry has fields such as _SYSTEMD_UNIT, _PID, PRIORITY, SYSLOG_IDENTIFIER and MESSAGE, which is why filtering by unit is exact instead of text search.

Persistence and size

By default on many distributions the journal lives in /run/log/journal and disappears at reboot, which is exactly when you need it. Make it persistent:

Terminal
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

or set Storage=persistent in /etc/systemd/journald.conf. In the same file limit the size:

TEXT
[Journal]
Storage=persistent
SystemMaxUse=500M
SystemKeepFree=1G
MaxRetentionSec=1month
Compress=yes

Clean up manually with sudo journalctl --vacuum-size=200M or --vacuum-time=2weeks.

Who may read it? Root, and members of the systemd-journal (and on Debian adm) groups. Add an operator with sudo usermod -aG systemd-journal alice rather than giving full sudo.

Priorities and facilities

Syslog (RFC 5424) labels every message with a severity (priority) number and a facility (the kind of source).

Number Priority Meaning
0 emerg system unusable
1 alert act immediately
2 crit critical condition
3 err error
4 warning warning
5 notice normal but significant
6 info informational
7 debug debug detail

Lower number means more severe. journalctl -p err shows 0 to 3. Common facilities are kern, user, mail, daemon, auth/authpriv (logins, sudo), cron, syslog, and local0 to local7 for your own applications.

Python Filter log lines by priority and unit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The classic /var/log files

Even on journal-first systems, text logs remain common, written by rsyslog or by applications themselves.

File Contents
/var/log/syslog (Debian) or /var/log/messages (RHEL) general system messages
/var/log/auth.log (Debian) or /var/log/secure (RHEL) logins, sudo, SSH
/var/log/kern.log kernel messages
/var/log/dmesg boot hardware messages
/var/log/cron, /var/log/mail.log scheduler and mail
/var/log/apt/, /var/log/dnf.log package manager history
/var/log/nginx/, /var/log/apache2/ web server access and error logs
/var/log/wtmp, btmp, lastlog binary login records, see last, lastb, lastlog

Handy tools: tail -F file (follow, and survive rotation), less +F file, grep -i error file, zgrep for compressed rotated files, and last -n 10 for recent logins.

rsyslog

rsyslog reads messages (from the journal or the /dev/log socket), applies rules, and writes files or forwards. Rules live in /etc/rsyslog.conf and /etc/rsyslog.d/*.conf. The classic selector format is facility.priority destination:

TEXT
auth,authpriv.*              /var/log/auth.log
*.*;auth,authpriv.none       -/var/log/syslog
kern.*                       -/var/log/kern.log
mail.err                     /var/log/mail.err
*.emerg                      :omusrmsg:*
local5.*                     /var/log/myapp.log

The leading - used to mean "do not sync after each write" (still accepted). auth.none excludes a facility.

Sending logs to a central server

Central logging protects evidence from tampering and lets you search all servers in one place. On the receiver, enable a listener (TCP is more reliable than UDP):

TEXT
# /etc/rsyslog.d/10-receive.conf  (log server)
module(load="imtcp")
input(type="imtcp" port="514")
template(name="PerHost" type="string" string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")
*.* ?PerHost

On each client:

TEXT
# /etc/rsyslog.d/90-forward.conf
*.* action(type="omfwd" target="logs.example.org" port="514" protocol="tcp"
           action.resumeRetryCount="-1" queue.type="LinkedList" queue.size="10000")

The queue keeps messages while the server is unreachable. For encryption use omfwd with TLS (StreamDriver="gtls"), because plain syslog sends text across the network. Test the pipeline with logger:

Terminal
logger -p local5.info -t myapp "deploy finished"
journalctl -t myapp -n 1

Write to the log from your own scripts the same way: logger -t backup "completed $(date)" is far better than appending to a private file nobody rotates. Larger setups ship logs with agents such as Vector, Fluent Bit, or Promtail to Loki, Elasticsearch or a hosted service; the principles (structured fields, central copy, retention) stay the same.

Where to look

journalctl -u unit One service.
auth.log or secure Logins and sudo.
journalctl -k Kernel, OOM, disks.
App logs /var/log/nginx and friends.

Make the journal persistent first.

1 / 2

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which command follows new log entries of the nginx service?

    -f follows, -u selects a systemd unit.
  2. 2. Why configure Storage=persistent for journald?

    After a crash and reboot you want the previous boot's logs: journalctl -b -1.
  3. 3. journalctl -p err shows which messages?

    -p LEVEL includes that level and everything more severe.
  4. 4. What is the main benefit of forwarding logs to a central server?

    An intruder on one host cannot erase the central copy.

Summary

  • journald stores structured entries; journalctl filters by unit, priority, boot, time, PID and text.
  • Make the journal persistent and cap its size with journald.conf and --vacuum-*.
  • Syslog severities run from 0 (emerg) to 7 (debug); facilities label the source.
  • Classic text logs in /var/log remain important: auth.log or secure, syslog or messages, application logs.
  • rsyslog rules route messages to files or to a central server, ideally over TCP with TLS and a disk or memory queue.

Next: Log Rotation and Retention