Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Deployment and Automation

Logging, Monitoring and the Hardware Watchdog

Keep an unattended Pi healthy: Python's logging module with rotation, journald, heartbeat checks, the hardware watchdog that reboots a frozen Pi, and simple uptime alerts.

A Pi that runs unattended must tell you what it is doing, and recover when something goes wrong. If it freezes in a greenhouse a hundred kilometres away, nobody will press the reset button. Three tools make a project robust: logs that explain what happened, monitoring that notices when it stops, and a watchdog that reboots a frozen board automatically.

The hardware watchdog and journald exist only on a real Pi, so those commands are read-only. Python's logging module and the monitoring logic run here.

Logging in Python

print is fine for experiments. For services use the logging module: it adds timestamps, severity levels and handlers that send messages to the console, a file or the journal.

Python
import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s: %(message)s",
)
log = logging.getLogger("weather")
log.info("started")
log.warning("sensor slow: %.2f s", 1.4)
log.error("read failed", exc_info=True)   # includes the traceback

Levels from quiet to loud: DEBUG, INFO, WARNING, ERROR, CRITICAL. Set the level to INFO normally and DEBUG while diagnosing. When the program runs under systemd, logging to stderr is enough: the journal stores it with timestamps (see systemd services).

Python Levels, formatting and filtering

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Log files and rotation

If you log to a file, rotate it, or it will grow until the card is full. RotatingFileHandler keeps a fixed number of old files of limited size:

Python
from logging.handlers import RotatingFileHandler

handler = RotatingFileHandler("/home/ana/weather/app.log",
                              maxBytes=500_000, backupCount=3)

That caps the log at about 2 MB (the live file plus three backups), safe for any card. The Linux logrotate tool does the same for files written by other programs. The model below shows the rotation algorithm, so you can see exactly how files are renamed.

Python Stand-in: size-based log rotation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The oldest lines (events 1 to 3 or so) are gone: rotation trades history for a bounded size.

Reading the system journal

Terminal
journalctl -u weather -n 50 --no-pager     # last 50 lines
journalctl -p warning -b                    # warnings and worse since boot
journalctl -k | grep -i voltage             # kernel messages, e.g. under-voltage
sudo journalctl --vacuum-time=14d           # delete older than 14 days

The kernel log is also where you see USB resets, filesystem errors and SD card problems. If a Pi reboots by itself, look at journalctl -b -1 (the previous boot) to read its last words.

Detecting that a program stopped

A program that crashes leaves a clear error, but one that hangs says nothing. Monitor it with a heartbeat: the program writes the current time to a file or sends an MQTT message every minute, and a checker complains when the heartbeat gets old.

Python Stand-in: heartbeat monitoring

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For free external monitoring, call a "dead man's switch" service (such as Healthchecks.io) with a simple HTTP request at the end of each job. If the ping stops, the service emails you. This also detects a dead Pi or lost internet, which a local script cannot.

Terminal
curl -fsS --retry 3 https://hc-ping.com/your-uuid-here > /dev/null

The hardware watchdog

The SoC has a hardware timer. When the watchdog is enabled, something must "pet" it periodically, and if it stops (kernel hang, overloaded system) the chip resets the Pi. It is the last line of defence for remote installations.

Let systemd manage it. In /etc/systemd/system.conf:

TEXT
RuntimeWatchdogSec=15
RebootWatchdogSec=2min

Systemd then feeds the watchdog every few seconds; if the whole system freezes, the board reboots after 15 seconds. Enable the device with dtparam=watchdog=on in /boot/firmware/config.txt on images where it is off (the SoC driver is usually loaded already, check ls /dev/watchdog*).

You can also make a single service watch itself: add WatchdogSec=60 to the unit and have the program send systemd.daemon.notify("WATCHDOG=1") regularly. If it hangs, systemd kills and restarts it according to Restart=.

flowchart LR
    A["Program loop"] -->|"pet every 20 s"| B["systemd"]
    B -->|"pet every few s"| C["Hardware watchdog"]
    B -.->|"missed pets"| D["Restart service"]
    C -.->|"missed pets"| E["Reboot Pi"]

Two levels of protection: restart a hung service, reboot a hung system.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use the logging module instead of print in a service?

    Levels and handlers let you control detail and where messages go.
  2. 2. What does RotatingFileHandler(maxBytes=500000, backupCount=3) protect against?

    It caps total log size by renaming old files and discarding the oldest.
  3. 3. Which command shows the logs of the previous boot?

    -b -1 selects the boot before the current one.
  4. 4. What does the hardware watchdog do when it is no longer fed?

    A missed deadline triggers a hardware reset, recovering from a total hang.

Summary

  • Use logging with levels; under systemd, stderr output lands in the journal.
  • Rotate log files so they cannot fill the card.
  • Use journalctl for services, the kernel and previous boots.
  • A heartbeat plus an external dead man's switch catches hangs and outages.
  • Enable the watchdog (RuntimeWatchdogSec) so a frozen Pi reboots itself.

Next: Project: A Data Logger with a Live Dashboard