Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 78 ยท Pods

Liveness, Readiness and Startup Probes

Learn the three Kubernetes probes, the HTTP, TCP, exec and gRPC probe types, how thresholds and delays work, and the mistakes that cause restart loops.

A running container is not necessarily a working one. A web server can be alive but still loading its cache, a worker can deadlock while its process keeps existing, and a database client can lose its connection and return errors forever. Kubernetes cannot see inside your program, so it asks. A probe is a small check that the kubelet runs against a container on a schedule, and the answer drives what Kubernetes does next.

There are three kinds of probe, and each answers a different question. A liveness probe asks "should this container be restarted?". A readiness probe asks "should this Pod receive traffic right now?". A startup probe asks "has the application finished starting?". Mixing these up is the most common cause of outages that Kubernetes itself creates. This lesson explains each one, the four ways to probe, how the timing settings interact, and then lets you watch a probe simulator make the decisions. If you used HEALTHCHECK in Docker, you will see how probes are more precise (Docker healthchecks).

Manifests here are read-only. The Python programs model the kubelet's counting rules so you can experiment with the settings. They do not run real probes.

Three probes, three questions

Liveness Is it stuck? If so, restart it.
Readiness Can it take traffic now?
Startup Has it finished booting?
1 / 2

The four probe mechanisms

Whatever the question, a probe must run one of four checks. You pick one per probe:

YAML
# 1. HTTP: any status from 200 to 399 is success
readinessProbe:
  httpGet:
    path: /healthz
    port: 8080

# 2. TCP: success if the port accepts a connection
livenessProbe:
  tcpSocket:
    port: 5432

# 3. Exec: success if the command exits with 0
livenessProbe:
  exec:
    command: ["sh", "-c", "test -f /tmp/alive"]

# 4. gRPC: success if the standard health service answers SERVING
readinessProbe:
  grpc:
    port: 9000

HTTP is the usual choice for web services. The kubelet sends the request itself, from the node, to the Pod's IP. TCP only proves that something is listening, which is weak but needs no code in your app. Exec runs a command inside the container each time, which costs CPU and a process spawn per check, so keep it light. gRPC needs the gRPC health checking protocol implemented in the app.

The timing settings

Every probe shares the same knobs:

Field Default Meaning
initialDelaySeconds 0 Wait this long after the container starts before the first probe.
periodSeconds 10 How often to probe.
timeoutSeconds 1 How long to wait for one answer before counting a failure.
failureThreshold 3 Consecutive failures needed to declare failure.
successThreshold 1 Consecutive successes needed to declare success (must be 1 for liveness and startup).

A single bad answer does nothing. The kubelet counts consecutive results. With the defaults, a probe has to fail three times in a row, ten seconds apart, so the verdict takes about 30 seconds. Raising failureThreshold makes the probe more patient; lowering it makes it twitchy.

One important detail: timeoutSeconds: 1 is short. An endpoint that does a database query and sometimes takes 1.5 seconds will count as failed, and a loaded server might fail its own liveness checks and get restarted exactly when it needs to keep working.

Liveness: restart a stuck process

Use liveness only for failures that a restart can fix, such as a deadlock or an event loop that stopped. A liveness endpoint should be cheap and should check only the process itself.

YAML
livenessProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

When it fails, the kubelet kills the container and applies the Pod's restartPolicy. The Pod keeps its name, node and IP, and the RESTARTS count goes up. Repeated restarts end in CrashLoopBackOff (Pod Lifecycle).

The biggest mistake is making liveness depend on things outside the container. If /livez calls the database and the database has a hiccup, every Pod fails liveness at the same moment, every Pod restarts and now the whole service is down while a recoverable problem is made worse.

Readiness: control traffic

A readiness failure does not restart anything. The kubelet marks the Pod NotReady, and the Pod's address is removed from the endpoints of every Service that selects it (Services). When the probe passes again, the Pod is added back. This is how rolling updates stay safe too: a new Pod only counts as available once it is ready (Rolling Updates and Rollbacks).

Readiness is the right place to check dependencies: "can I reach the database?", "has the cache warmed up?". It is also fine for a Pod to become unready temporarily when it is overloaded, to shed load.

YAML
readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 5
  failureThreshold: 2
  successThreshold: 1

Without a readiness probe a container counts as ready the moment it starts. That means a slow-starting app receives traffic before it can answer, and users see errors during every deploy.

Startup: protect slow starters

Some applications need a long time to boot, such as a JVM service loading a large model. If liveness started probing at second 10, it would kill the app again and again before it finished. You could use a huge initialDelaySeconds, but then a real deadlock after startup is noticed very late.

A startup probe solves this. While it has not yet succeeded, liveness and readiness probes are disabled. It gets a generous budget of failureThreshold * periodSeconds, and once it succeeds once, it stops and the other two take over with their normal tight settings.

YAML
startupProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 5
  failureThreshold: 60      # up to 300 seconds to start
livenessProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

If the startup probe never succeeds within its budget, the container is killed and restarted, just like a liveness failure.

A probe simulator

This stand-in plays the kubelet's role against a pretend application. The app boots in a given number of seconds, and may hang at some moment. The simulator ticks once per second and prints what happens: when the Pod becomes ready, and when a liveness failure causes a restart. Try changing the numbers.

Python Probes deciding readiness and restarts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the four scenarios as short stories. In A, the readiness probe is the first to pass once the app answers. In B, liveness starts at once, sees no answers during boot and kills the app three probes later, so it never finishes starting and restarts in a loop. In C, the startup probe absorbs the long boot and then hands over. In D, the app first becomes unready (traffic stops), and later liveness decides the process is dead and restarts it.

Choosing thresholds

Probe settings are a trade-off between how quickly you notice a problem and how often you react to a false alarm. The helper below computes the worst-case time to detect a failure and the time a failing Pod stays in service, for a few settings. Time to detect is roughly failureThreshold * periodSeconds plus up to one more period before the first failing check.

Python How long until a probe notices?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Liveness checks dependencies. A database outage restarts every app Pod. Keep liveness local to the process.
  • Same endpoint and same settings for liveness and readiness. They have different jobs. Often readiness checks more things and runs more often.
  • No readiness probe. Traffic arrives before the app can serve it, which makes every rollout noisy.
  • Slow starter without a startup probe. The app is killed before it finishes booting.
  • Timeout too short. Under load the probe itself times out, and Kubernetes restarts a Pod that was only busy.
  • Probing with exec and heavy commands. The cost multiplies by every Pod and every period.
  • Probes that always return 200. A check that cannot fail protects nothing.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A readiness probe fails for a Pod. What does Kubernetes do?

    Readiness only controls traffic. The container keeps running, and the Pod is added back to the endpoints once the probe succeeds.
  2. 2. Why is it risky for a liveness probe to call the database?

    Liveness should reflect only whether the process itself is stuck. A shared dependency failing is a readiness concern, not a reason to restart healthy processes.
  3. 3. What does a startup probe do while it has not yet succeeded?

    Liveness and readiness wait until the startup probe has succeeded once, giving slow applications time to boot without being killed.
  4. 4. With periodSeconds: 10 and failureThreshold: 3, roughly how long until a broken container is declared failed?

    Three consecutive failures ten seconds apart take about 30 seconds, plus up to one more period before the first failing check.
  5. 5. Which HTTP status codes count as success for an httpGet probe?

    The kubelet treats any status of at least 200 and below 400 as success. Anything else, or a timeout, is a failure.

Summary

  • Liveness restarts a stuck container, readiness controls whether the Pod gets traffic, and startup protects slow boots.
  • Probe mechanisms are httpGet, tcpSocket, exec and grpc.
  • The kubelet counts consecutive results against failureThreshold and successThreshold every periodSeconds, with a timeoutSeconds per attempt.
  • Keep liveness cheap and local, put dependency checks in readiness, and use a startup probe for slow applications.
  • Most probe-related outages come from liveness probes that are too strict, too dependent or too early.

Next, learn how Pods ask for CPU and memory in Resource Requests and Limits.