Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 37 ยท Monitoring and Logs

Log Rotation and Retention with logrotate

Keep logs from filling the disk: how logrotate works, writing config files, rotate and compress options, postrotate scripts, copytruncate, testing with -d and retention planning.

A busy web server can write gigabytes of access logs a week. Left alone, they fill the disk, and a full disk stops databases, breaks logins and corrupts state. Log rotation closes the current log file, renames it, compresses old copies, and starts a new one, deleting the oldest after a set number of cycles. On Linux the standard tool is logrotate, run once a day by a systemd timer (or /etc/cron.daily/logrotate on older systems).

This lesson teaches how to read and write logrotate configuration, how to test it safely, and how to decide retention. Real logrotate needs real files, so commands are read-only; the Python stand-ins simulate the rotation naming and the "how much space do I need" calculation.

How logrotate is organised

  • /etc/logrotate.conf: global defaults and an include /etc/logrotate.d line.
  • /etc/logrotate.d/*: one file per application (nginx, apache2, rsyslog, apt, your own app). Packages ship their own.
  • /var/lib/logrotate/status (or /var/lib/logrotate.status): when each file was last rotated.
  • The schedule: systemctl list-timers logrotate.timer. It fires daily, but each file's own rule (weekly, monthly, size) decides whether it is actually rotated that day.

A typical application stanza:

TEXT
/var/log/myapp/*.log {
    daily
    rotate 14
    missingok
    notifempty
    compress
    delaycompress
    dateext
    create 0640 myapp adm
    sharedscripts
    postrotate
        systemctl reload myapp >/dev/null 2>&1 || true
    endscript
}

The directives you need

Directive Meaning
daily, weekly, monthly, yearly how often to rotate
size 100M rotate when the file exceeds this size (replaces the time rule)
maxsize 100M rotate on schedule, or sooner if larger
rotate N keep N rotated copies; older ones are removed
maxage N remove rotated files older than N days
compress / delaycompress gzip rotated files; delay compression one cycle (so a process still writing to the old file is safe)
compresscmd, compressext choose zstd or xz instead of gzip
dateext name old files with a date (app.log-20251112) instead of .1, .2
missingok no error if the log does not exist
notifempty skip empty logs
create MODE USER GROUP create a new empty log with these permissions after rotation
copytruncate copy the log, then truncate the original in place
su USER GROUP rotate as this user (needed when the directory is not root-owned)
sharedscripts run the scripts once for all matching files, not once per file
prerotate / postrotate ... endscript commands to run around rotation

The "program keeps writing to the old file" problem

Rotation renames app.log to app.log.1. A running program still holds the old file open (by file descriptor), so it keeps writing to app.log.1, and the new app.log stays empty. There are two cures:

  1. Tell the program to reopen its logs in postrotate (nginx -s reopen, systemctl reload nginx, kill -HUP $(cat /run/app.pid), rsyslog-rotate). This is the clean approach.
  2. copytruncate: copy the contents to the rotated file, then empty the original. No signal is needed, but lines written between the copy and the truncate are lost, and large files cost extra I/O. Use it only for programs that cannot reopen files.

Programs that log through journald or syslog (see journald and rsyslog) do not need logrotate for those messages: the journal has its own size and age limits.

Python Simulate rotation file names

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Testing before you trust it

Never write a logrotate rule and wait a week to find out it is wrong.

Terminal
sudo logrotate -d /etc/logrotate.d/myapp       # debug: show what WOULD happen, change nothing
sudo logrotate -f /etc/logrotate.d/myapp       # force a rotation now
sudo logrotate -v /etc/logrotate.conf          # verbose real run
cat /var/lib/logrotate/status | head

Common errors you will see:

  • error: skipping "..." because parent directory has insecure permissions : the log directory is writable by a group or others; fix permissions or add a su user group line.
  • ...Permission denied : the postrotate command lacks the rights, or SELinux/AppArmor blocks it (see MAC).
  • The log is rotated but the application keeps filling the old file: it never reopened (see above).
  • Files grow past the limit between runs: with a daily timer, size rules are only checked once a day. Use maxsize or run logrotate hourly (/etc/cron.hourly or a timer override).

Planning retention

Retention balances four forces: disk space, troubleshooting needs, legal and compliance rules (for example you may be required to keep authentication logs for a year), and privacy rules (IP addresses are personal data under GDPR, so do not keep access logs longer than you can justify). A pragmatic scheme: keep a few days of full detail locally for fast searching, and ship everything to a central, cheaper store for the long term (compressed, perhaps in object storage), where the local copy can then be short.

Estimate space before you choose a number: daily average size x copies x compression factor. Text logs compress to about 10 to 20 percent with gzip.

Python How much disk will my logs use?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Add a monitoring alert for /var/log or /var usage above 80 percent (see Metrics and Alerting). Rotation silently stopping is one of the most common causes of "disk full" incidents.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does delaycompress do?

    A program that still writes to the just-rotated file is safe, because compression happens one cycle later.
  2. 2. When is copytruncate the right choice?

    It avoids signalling but may lose lines written during the copy, so prefer a reload in postrotate.
  3. 3. Which command shows what logrotate would do without changing anything?

    -d is debug (dry-run) mode and prints the planned actions.
  4. 4. A size 100M rule is not triggering often enough. Why?

    Run logrotate more often or use maxsize with an hourly timer.

Summary

  • logrotate is driven by a daily timer; per-application stanzas in /etc/logrotate.d choose frequency, count, compression and scripts.
  • Make the program reopen its log in postrotate; use copytruncate only as a fallback.
  • Test with logrotate -d, force with -f, and read the status file.
  • Plan retention from disk size, troubleshooting needs, compliance and privacy; keep long-term copies on central storage.
  • Alert on /var usage so a broken rotation is noticed before the disk fills.

Next: Performance Monitoring Tools