Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 111 ยท Containers in Depth

Checkpoint/Restore, live-restore and the Live Migration Myth

Learn what docker checkpoint (CRIU) can and cannot do, how daemon live-restore keeps containers alive during upgrades, and why containers do not live-migrate.

Sooner or later someone asks: "Can I move a running container to another server without stopping it?" or "Can I upgrade Docker without killing my containers?" The answers are not really and yes, with a setting, and the details are worth knowing, because the words "checkpoint", "pause" and "restart" sound similar and do very different things.

This lesson separates four ideas: pausing a container, checkpointing it with CRIU, keeping containers alive while the daemon restarts (live-restore), and the live migration that people expect from virtual machines. Commands were run on Docker 29.8.1.

Process state is memory, and memory is not in the image

A container is an ordinary Linux process (or several) with namespaces and cgroups around it (Namespaces and Cgroups). Everything the program knows at this moment lives in RAM: variables, open sockets, half-processed requests. None of that is part of the image or the writable layer, so tools that save the filesystem do not save the running program.

You can see this with a container that counts in memory:

Terminal
docker run -d --name ticker busybox sh -c 'i=0; while true; do i=$((i+1)); echo tick $i; sleep 1; done'
docker restart -t 1 ticker
docker logs --since 4s ticker

After the restart the counter starts again from 1. A restart is a stop followed by a fresh start; the old process and its memory are gone. docker commit would not help either, because it saves files, not process memory (Container Lifecycle).

docker pause: frozen, not saved

docker pause puts the container's processes in the cgroup freezer. They stay in RAM exactly as they were, they just get no CPU time:

Terminal
docker pause ticker
docker inspect ticker --format 'paused={{.State.Paused}} status={{.State.Status}}'
sleep 3
docker unpause ticker
TEXT
paused=true status=paused

After unpause the counter continues from the same number, with no restart. That is real "resume" behavior, but the state only survives as long as the daemon, the host and the container stay up. Pausing does not write anything to disk, so it is not a backup and it does not survive a reboot.

docker checkpoint: saving process state with CRIU

CRIU (Checkpoint/Restore In Userspace) is a Linux tool that can freeze a process tree, write its memory, open files and sockets to a directory of image files, and later rebuild the same processes from them. Docker exposes it through docker checkpoint, which is an experimental feature:

Terminal
docker checkpoint create ticker cp1        # save state, stop the container
docker checkpoint ls ticker                # list saved checkpoints
docker start --checkpoint cp1 ticker       # restore from cp1
docker checkpoint rm ticker cp1

Add --leave-running to create if the container should keep running after the dump. Checkpoints are stored next to the container under /var/lib/docker unless you pass --checkpoint-dir.

Whether it works depends on your setup. On the test machine (Docker Desktop, experimental features off) the command simply refuses:

TEXT
$ docker checkpoint create ticker cp1
docker checkpoint create is only supported on a Docker daemon with experimental features enabled

To try it on a Linux host you need three things: experimental mode ("experimental": true in /etc/docker/daemon.json), CRIU installed on the host, and a runtime that supports checkpointing (runc does; the Desktop VM does not ship CRIU for you). Even then there are sharp edges:

  • Network connections. Established TCP connections are hard to restore, and the container's IP address may be different on the target. Servers with many clients usually lose them.
  • Kernel dependence. The restored process expects the same kernel features and CPU instruction set as before. Restoring on a different kernel version or a different CPU model can fail.
  • Unsupported resources. Some devices, namespaces and mounts cannot be dumped. TTYs and some file descriptors need special flags.
  • Docker coverage. docker checkpoint works for one container on one host. It is not a cluster feature and it is not part of the stable API promise.

Good uses are narrow: warm-starting a slow-booting JVM or ML process, debugging a process by freezing and copying its state, and research. For production recovery, restart from a known image and keep state in volumes (Volumes, Backup and Restore).

Kubernetes has a similar alpha feature, container checkpointing, built on CRIU too. It has the same limits and is meant for forensic analysis, not for moving workloads between nodes.

live-restore: restarting the daemon without stopping containers

The second question is about upgrades. By default, stopping dockerd also stops every container, because the daemon is the parent that manages them. That makes a security patch for Docker a short outage.

Live restore changes that. With it enabled, containers keep running while the daemon is down, and the daemon reconnects to them when it starts again:

JSON
{
  "live-restore": true
}

You set it in /etc/docker/daemon.json (Securing the Docker Daemon) and reload or restart the daemon once. Docker reports the state in docker info, where it is off by default:

TEXT
$ docker info --format 'live-restore: {{.LiveRestoreEnabled}}'
live-restore: false

What it does not cover:

  • Swarm mode. live-restore is not compatible with Swarm.
  • Major daemon upgrades between some versions, or changes to the daemon's networking options, may need containers to be recreated. Read the release notes first.
  • Logs and exec while the daemon is down. The container's output goes to a pipe that the daemon normally empties. If the daemon stays away for long and the container writes a lot, the pipe fills and the container can block on its next write.
  • Management. Nothing can start, stop, or inspect containers until the daemon is back. Restart policies do not apply while it is down.
  • Port publishing and networking rules stay in the kernel and keep working, but new connections to new containers obviously cannot be set up.

So live-restore turns a "restart dockerd" into a short blind spot rather than an outage. It is a good default for single servers, and it is the reason a Docker package update on a well-configured host does not wake you up at night.

Why containers do not live-migrate

Virtual machines can move between hosts with almost no pause, because the hypervisor controls the whole virtual machine: CPU registers, memory pages, and a virtual disk on shared storage. It copies memory pages while the VM keeps running, then switches over.

Containers share the host kernel, so a "container" is spread across kernel objects: PIDs, cgroups, network namespaces, iptables rules, mounts, file locks. CRIU can rebuild some of them, but the kernel never stops caring about hardware details, kernel versions and where its sockets live. There is no supported "copy the pages, keep the sockets" path. Instead, the container world solves the same business need differently:

Goal Container answer
Move a workload to another host Start a new container there from the same image; stop the old one. Keep state in a volume or external database.
Zero-downtime upgrade Run two copies behind a proxy or load balancer and shift traffic (Reverse Proxy project); Compose, Swarm and Kubernetes do rolling updates.
Survive a node failure Restart policies on one host (Container Lifecycle), an orchestrator across hosts (Orchestration Basics).
Fast start for heavy runtimes Smaller images, warm pools of containers, or CRIU where you accept its limits.
Real live migration Use VMs (or run containers inside VMs that migrate).

The design rule that follows: treat containers as replaceable. Do not hide irreplaceable state inside one, and you will never need to migrate it.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A container keeps a counter only in memory. What happens to the counter after docker restart?

    A restart is a stop plus a start. The old process and its memory are gone, and neither the image nor the writable layer contains process memory.
  2. 2. What is the main difference between docker pause and docker checkpoint create?

    Pause uses the cgroup freezer and keeps everything in memory, so it does not survive a reboot. Checkpoint dumps the process state to files, but it is experimental and needs CRIU.
  3. 3. On the test machine, docker checkpoint create printed an error. Why?

    The command is only available on a daemon with experimental features enabled, and it also needs CRIU on the host.
  4. 4. What does live-restore: true in daemon.json do?

    With live restore the containers survive a daemon restart, which makes patching dockerd much less disruptive. It is not compatible with Swarm mode.
  5. 5. Which approach is the container-friendly way to move a service to another server?

    Containers are replaceable. Start a fresh one elsewhere and keep the state outside the container, so no process memory has to move.

Summary

  • Process memory is not saved by images, docker commit or docker restart; a restarted container starts fresh.
  • docker pause freezes processes in RAM (cgroup freezer). It is cheap and reversible, but not persistent.
  • docker checkpoint uses CRIU to dump and restore a process tree. It is experimental, needs the right host setup, and struggles with network connections and kernel differences.
  • live-restore: true keeps containers running while dockerd restarts. It is off by default and incompatible with Swarm.
  • Containers do not live-migrate like VMs. Replace instead of move: new container, same image, state in volumes or external services.

Next: Docker Metrics, Tracing and Dashboards shows how to watch what your containers are doing over time.