Skip to content
Tutorials Center
My learning

Topics

Lesson 72 of 149 ยท Files and Errors

Logging and Warnings

Set up real logging in Python with loggers, handlers, formatters and dictConfig, log to files, and use the warnings module for deprecations and soft problems.

In Debugging and assert you met the logging module as a better print(): messages with levels that you can switch on and off with one setting. That's enough for a small script. Real applications need more: messages on screen and in a file at the same time, timestamps, different detail levels for different parts of the program, and libraries that log without forcing their output on you.

This lesson shows how logging is really organized (loggers, handlers and formatters), how to configure it for a whole application, and how to log errors properly. Then you'll meet its sibling, the warnings module, which is how Python and libraries tell you "this still works, but you should change it".

Three building blocks

Every log message passes through three kinds of objects:

  • A logger is what your code calls: log.info("..."). Loggers have names like "shop.orders" and form a tree.
  • A handler decides where messages go: the console, a file, an email, a log server. One logger can have several handlers.
  • A formatter decides what each line looks like: timestamp, level, logger name, message.

Both loggers and handlers have a level. A message must pass the logger's level first, then each handler's level. That's how you can show only warnings on screen while a file records every detail.

Python A logger with two handlers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The console showed two lines, while the file recorded all four with timestamps and line numbers. Notice the log.handlers.clear() line: addHandler() adds a new handler every time it runs, and duplicate handlers print every message twice. In a real program you configure logging once at startup, so you won't need that line. The editors on this page keep the logging module's settings between runs (even across examples), so the demos reset what they change. log.propagate = False keeps this logger's messages from also reaching the root logger; you'll see why that matters in the hierarchy section.

Formatting log lines

A formatter's format string uses %(name)s-style placeholders for the attributes of each log record. The ones you'll use most:

Placeholder Meaning
%(asctime)s time of the message (shape it with datefmt)
%(levelname)s DEBUG, INFO, WARNING, ERROR or CRITICAL
%(name)s the logger's name
%(message)s the message, with its arguments filled in
%(module)s, %(funcName)s, %(lineno)d where the call happened
%(process)d, %(threadName)s process ID and thread name

You can write -8 after the % to pad a field (%(levelname)-8s), which keeps columns aligned. If you prefer {} placeholders, pass style="{": Formatter("{asctime} {levelname} {message}", style="{").

Keep passing values as arguments (log.info("total %s", total)) rather than pre-formatting with f-strings. The text is only built if a handler actually outputs the message, and tools that collect logs can group messages that share the same template.

The logger hierarchy

Logger names with dots form a tree. "shop.orders" is a child of "shop", which is a child of the root logger. When a logger has no level of its own, it uses its parent's; and after a logger's own handlers run, the message propagates up to the handlers of its ancestors:

Python Loggers form a tree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This design is why the convention is one logger per module, created with log = logging.getLogger(__name__). A module shop/orders.py gets the logger "shop.orders" automatically, the output shows which module each message came from, and you can raise or lower the detail of any package without touching its code. Usually only the root logger has handlers, and every other logger just propagates to it.

Logging errors

When something fails, the traceback is the most valuable information you have. Log it, don't just log the message:

Python Logging exceptions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

log.exception() only belongs inside an except block. Elsewhere, any level method accepts exc_info=True to attach the current traceback, and stack_info=True to show how the code got there. The extra={...} dictionary adds custom fields to the record, which formatters can show (%(rows)s) and log collectors can index.

Configuring a whole application

basicConfig() covers the common case: one handler on the root logger. Call it once, at the start of your program, and nowhere else. For anything bigger, describe the setup as data with logging.config.dictConfig(). The dictionary could just as well come from a JSON, YAML or TOML file:

Python dictConfig

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

dictConfig() replaces the current configuration, so it's also safe to rerun. Keep "disable_existing_loggers": False; otherwise loggers that modules created at import time are silenced, which surprises everyone once.

Rotating log files

A log file that only grows will eventually fill the disk. The logging.handlers module has handlers that start a new file by size or by time and keep a few old ones:

Python
from logging.handlers import RotatingFileHandler, TimedRotatingFileHandler

# At most 5 files of 1 MB: app.log, app.log.1 ... app.log.4
by_size = RotatingFileHandler("app.log", maxBytes=1_000_000, backupCount=4, encoding="utf-8")

# A new file every midnight, keeping two weeks
by_day = TimedRotatingFileHandler("app.log", when="midnight", backupCount=14, encoding="utf-8")

On servers and in containers, the usual practice is simpler still: log to standard output and let the platform collect the logs.

Logging in libraries

If you write a library that others import, create module loggers with getLogger(__name__) as usual, but never call basicConfig() or add console handlers: that's the application's decision. Add a NullHandler to your package's top logger so nothing is printed unless the application configures logging:

Python
# mylib/__init__.py
import logging

logging.getLogger(__name__).addHandler(logging.NullHandler())

The warnings module

Some problems aren't errors. A function still works but will be removed in the next version; an argument combination is allowed but probably a mistake. For these, Python has warnings. A warning prints a message (once per code location by default) and lets the program continue:

Python Issuing warnings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The warning text goes to standard error, so it may appear slightly apart from your printed output. Each warning has a category, a subclass of Warning:

Category Use it for
UserWarning the default; general advice to the caller
DeprecationWarning features that will be removed, aimed at developers
FutureWarning behavior that will change, aimed at end users of a library
RuntimeWarning dubious runtime situations
ResourceWarning resources such as files that weren't closed

stacklevel=2 is worth a habit: it makes the warning report the line that called your function, which is the line the reader needs to change.

Note: By default Python hides DeprecationWarning unless it's triggered directly by code in your main script, so end users aren't bombarded by warnings from libraries. Test runners like pytest show them all. Since Python 3.13 you can also mark a function with the @warnings.deprecated("message") decorator, which warns at runtime and tells type checkers too.

Controlling warnings

Filters decide what happens to a warning: show it ("default", "always", "once"), hide it ("ignore"), or raise it as an exception ("error"). Change filters temporarily with warnings.catch_warnings(), which also lets tests record warnings:

Python Filtering, recording and escalating warnings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Outside your code, you can set filters for a whole run with the -W command-line option (python -W error::DeprecationWarning app.py) or the PYTHONWARNINGS environment variable. Running your test suite with deprecations turned into errors is a cheap way to prepare for library upgrades.

Warnings into logs

In a long-running application, warnings printed to standard error are easy to miss. logging.captureWarnings(True) reroutes them through the logger named "py.warnings", so they land in your log files with everything else.

Logging, warnings or exceptions?

  • Raise an exception when the code can't do what was asked. See Raising Exceptions.
  • Issue a warning when the caller should change their code, but it still works this time.
  • Log to record what happened while the program runs, for whoever operates it.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A logger's level is DEBUG and its console handler's level is WARNING. Which call appears on the console?

    Messages must pass both the logger's level and the handler's level, so only WARNING and above reach the console.
  2. 2. Why should each module create its logger with logging.getLogger(__name__)?

    The module name builds a dotted hierarchy like shop.orders, which shows up in output and can be configured at any level.
  3. 3. What is wrong with calling logging.basicConfig() inside a library you publish?

    Libraries create loggers and log to them. The application decides where messages go and how they look.
  4. 4. What does stacklevel=2 do in warnings.warn(..., stacklevel=2)?

    The caller's line is the code that needs to change, so pointing the warning there makes it actionable.
  5. 5. How can a test make every DeprecationWarning fail loudly?

    The "error" action turns matching warnings into exceptions, and catch_warnings restores the previous filters afterwards.

Summary

  • Loggers (what you call) send records to handlers (where they go), which use formatters (how they look); both loggers and handlers filter by level.
  • Create one logger per module with getLogger(__name__); dotted names form a tree, and messages propagate up to the root's handlers.
  • Configure once at startup: basicConfig() for simple programs, logging.config.dictConfig() for real applications; rotate files with RotatingFileHandler or TimedRotatingFileHandler.
  • Use log.exception() in except blocks to keep the traceback; pass values as arguments, not f-strings.
  • Libraries add a NullHandler and never configure logging.
  • warnings.warn(message, Category, stacklevel=2) reports soft problems; filters, catch_warnings(), -W and PYTHONWARNINGS control them.

Next: Classes and Objects, where you'll start object-oriented programming by designing your own types.