Skip to content
Tutorials Center
My learning

Topics

Lesson 58 of 78 ยท Operations and Observability

Logging in Kubernetes

Learn how Kubernetes logging works: stdout and stderr, kubectl logs, node-level agents, Loki and EFK, structured logs, plus a runnable log aggregator.

When something goes wrong in production, the first question is "what did the application say?" On a single server you would open a file under /var/log. In Kubernetes the same container may run on any node, be restarted, be rescheduled or be deleted in the middle of the night, taking its local files with it. Logging therefore needs a plan: how output leaves the container, where it is kept, and how you search it across hundreds of pods.

Kubernetes itself provides only the first part, a basic mechanism for capturing a container's output and reading it with kubectl logs. Everything beyond that (central storage, search, retention, alerts) comes from tools you add. This lesson explains the built-in part, the standard architecture for the rest, and how to write logs that are easy to query. It pairs with Metrics and Monitoring with Prometheus and the Troubleshooting Playbook.

A cluster and a log backend cannot run inside this page. All kubectl commands and agent configuration below are read-only, and the Python programs are stand-ins: they parse the node log format and run label queries in the style of Loki, but they are tiny models, not the real software.

The journey of a log line

App writes To stdout or stderr, one event per line.
Runtime saves The container runtime stores it in a file on the node.
Agent ships A DaemonSet reads the files and forwards them.
Backend searches Loki, Elasticsearch or a cloud service.
1 / 2

stdout and stderr

The Kubernetes convention, borrowed from the twelve-factor approach, is that an application writes its logs as an event stream to standard output and standard error and does not manage files, rotation or shipping itself. The container runtime (containerd or CRI-O) captures both streams. This is the same behavior you saw with docker logs in the Docker logging lesson.

On the node, the runtime writes each container's output to a file and the kubelet manages it:

TEXT
/var/log/pods/<namespace>_<pod>_<uid>/<container>/0.log
/var/log/containers/<pod>_<namespace>_<container>-<id>.log   (symlinks)

Each line uses the CRI log format: a timestamp, the stream, a tag and the message.

TEXT
2026-11-06T10:00:01.100000000Z stdout F {"level":"info","msg":"listening","port":8080}
2026-11-06T10:00:05.250000000Z stderr F panic: connection refused

The tag is F for a full line and P for a partial one: very long lines are split by the runtime and an agent must glue the pieces back together. The kubelet rotates these files (by default at 10 MiB, keeping five), so a chatty container cannot fill the disk forever. It also means node-local logs are short-lived: when a pod is deleted or evicted, its log files go with it.

kubectl logs

kubectl logs asks the API server, which asks the kubelet on the right node, to stream the file. It is your first tool for any problem.

Terminal
kubectl logs web-7d9f8c6b5-abc12                 # one pod
kubectl logs web-7d9f8c6b5-abc12 -c sidecar      # one container of a multi-container pod
kubectl logs -f web-7d9f8c6b5-abc12              # follow, like tail -f
kubectl logs --previous web-7d9f8c6b5-abc12      # the crashed earlier container
kubectl logs --since=10m --tail=100 --timestamps web-7d9f8c6b5-abc12
kubectl logs deploy/web                          # picks one pod of the Deployment
kubectl logs -l app=web --all-containers --prefix --tail=20   # many pods at once

Remember these limits. --previous is the way to read why a container in CrashLoopBackOff died (Pod Lifecycle explains the restarts). Only the current and the previous container instance are available, and once the pod object is gone so are its logs. Selecting with -l shows logs from a limited number of pods at a time. For anything bigger, or anything historical, you need a central system.

Node-level agents

The standard design runs a logging agent on every node as a DaemonSet. The agent mounts /var/log from the host, tails the container log files, enriches each line with metadata from the Kubernetes API (namespace, pod name, container, labels), and sends it to a backend. Popular agents are Fluent Bit (small and fast), Fluentd, Vector, Grafana Alloy (the successor of Promtail) and the OpenTelemetry Collector.

YAML
# Fluent Bit configuration (read-only), shipped to Loki
[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    multiline.parser  docker, cri
    Tag               kube.*

[FILTER]
    Name              kubernetes
    Match             kube.*
    Merge_Log         On          # parse JSON messages into fields

[OUTPUT]
    Name                    loki
    Match                   kube.*
    Host                    loki.monitoring.svc
    Port                    3100
    auto_kubernetes_labels  on
    labels                  job=fluent-bit

The agent approach needs no change to your applications and survives pod deletion, because lines leave the node within seconds. Two alternatives exist for special cases. A sidecar container can tail a log file an application insists on writing and print it to its own stdout (the adapter pattern from Init Containers and Sidecars). Or the application can send logs directly to a backend over the network, which is simple but ties your code to the backend and loses logs when the backend is down.

The first stand-in plays the agent. It reads CRI-format files, reconstructs partial lines, takes the namespace and pod from the file path, and attaches labels fetched from a (fake) API server.

Python A node-level log agent

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two partial lines were joined into one JSON event with the order number, the unstructured line is kept as plain text, and every record knows where it came from. That metadata is what makes cross-pod queries possible.

Backends: Loki, EFK and cloud services

Where do the lines end up? Three families dominate.

  • Grafana Loki indexes only the labels (namespace, pod, app) and stores the log text compressed in cheap object storage. It is inexpensive to run and integrates with Grafana, and you query it with LogQL, for example {app="web"} |= "timeout". The trade-off is that searching text means scanning chunks, so you narrow by labels first.
  • EFK or ELK (Elasticsearch, Fluent Bit or Logstash, Kibana; OpenSearch is the open-source fork) indexes the full text of every line. Searching is powerful and fast, but storage and memory cost much more and the cluster needs care.
  • Cloud services (CloudWatch, Cloud Logging, Azure Monitor, Datadog and others) take the same agents' output as a managed service. You trade control for convenience and a bill that grows with volume.

Whatever you choose, set a retention period (for example 14 days hot, 90 days cold) and keep an eye on volume: a debug log level left on in production can cost more than the application servers.

Structured logs

Free-text lines like User bob failed login from 10.0.0.5 after 3 tries are readable but hard to query. Prefer one JSON object per line:

JSON
{"ts":"2026-11-06T10:00:07Z","level":"error","msg":"payment timeout","order":1042,"trace_id":"4bf92f35"}

Now an agent or backend can filter on level, group by order, and join with traces through trace_id (see Tracing and Service Meshes). Keep each event on a single line (multi-line stack traces become separate events unless the agent has a multiline parser, or you put the stack in a field), use a small fixed set of levels, and never log secrets, tokens or personal data. A good rule: log events that a person or alert would act on, not every function call.

One caution for Loki: labels should have few distinct values. Namespace, app and pod are fine; a user id or request id as a label creates millions of tiny streams and ruins performance. Put those in the log line and filter on them at query time.

A query engine in miniature

The second stand-in implements the core of a LogQL-style query: a label selector, line filters (|= keeps lines containing text, != drops them), a | json stage that turns the message into fields, and a field filter.

Python Label queries over aggregated logs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the second query step by step: the selector narrows the search to the shop namespace, | json parses each message into fields, and level="error" keeps only matching events. The unstructured segfault line has no fields, so a field filter drops it; this is a reason to keep a consistent format and still alert on crash text separately. The final aggregation, counting errors per namespace, is what a dashboard panel does.

Cluster-level logs and good habits

Application logs are only part of the picture. Control-plane components log to files or the journal on their nodes, and the kubelet and container runtime run under systemd, so their logs are read with journalctl -u kubelet. Events (kubectl get events) record what controllers did and expire after about an hour by default; ship them to your backend if you want to see them later. The API server can also write an audit log of every request, covered in Supply Chain and Cluster Hardening.

A short checklist: write to stdout and stderr, use JSON with a stable schema, include a request or trace id, keep label cardinality low, set rotation and retention, and test your pipeline before an incident rather than during one.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Where should a containerized application normally write its logs in Kubernetes?

    The runtime captures stdout and stderr, so kubectl logs and node-level agents can read them without any application-side file handling.
  2. 2. A pod is in CrashLoopBackOff. Which command shows why the previous container instance died?

    --previous prints the logs of the last terminated instance of the container, which is where a crash message usually appears.
  3. 3. Why run the logging agent as a DaemonSet?

    Container logs live on the node that runs the pod, so each node needs its own agent, and a DaemonSet places one on every node.
  4. 4. What is a good Loki label?

    Loki indexes labels, so they must have few distinct values. App, namespace and pod are fine; unique values such as request ids belong in the log line.
  5. 5. What does the P tag in a CRI log line mean?

    Long lines are split by the runtime. P marks a fragment and F the final piece, and an agent joins them again.

Summary

  • Applications log to stdout and stderr; the runtime stores the streams in node files in CRI format and the kubelet rotates them.
  • kubectl logs reads those files for one pod or container (-f, --previous, --since, -l), but history disappears with the pod.
  • A DaemonSet agent such as Fluent Bit, Vector or Alloy tails the files, adds Kubernetes metadata and ships lines to a backend.
  • Loki indexes labels and stays cheap; Elasticsearch indexes full text and costs more; cloud services are managed alternatives.
  • Prefer one-line JSON events with stable fields and trace ids, keep label cardinality low, and set retention.

Next, add numbers to the story: Metrics and Monitoring with Prometheus.