Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 111 ยท Containers in Depth

Resource Limits

Limit memory, CPU, processes and open files in Docker containers. Learn --memory, --cpus, --pids-limit, ulimits, OOM kills (exit 137) and how cgroups enforce them.

By default a container may use as much CPU and memory as the host will give it. That is fine on your laptop and dangerous on a shared server: one runaway container with a memory leak or an infinite loop can starve every other container, and finally the host itself. Resource limits are the seat belts. They keep a misbehaving program inside its own box and make resource usage predictable.

This lesson shows how to set limits for memory, CPU, processes and open files; how to recognize the out-of-memory (OOM) killer at work; and what is happening underneath in the Linux kernel. Every output comes from a real run on Docker Engine 29.8 (Docker Desktop with a 6-CPU, ~12 GB virtual machine).

How limits work: cgroups

Docker does not enforce limits itself. It asks the Linux kernel to do it using control groups (cgroups). A cgroup is a labeled bucket of processes with settings such as "at most 64 MiB of memory". Docker creates one cgroup per container, writes your limits into it, and the kernel handles the rest. You can even read the settings from inside the container as small files:

CONSOLE
$ docker run --rm --memory 64m --cpus 0.5 --pids-limit 20 alpine:3.22 sh -c 'cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/cpu.max /sys/fs/cgroup/pids.max'
67108864
50000 100000
20
$ docker run --rm alpine:3.22 sh -c 'cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/cpu.max /sys/fs/cgroup/pids.max'
max
max 100000
max

The first run shows the limits as the kernel sees them: 67,108,864 bytes (64 MiB), a CPU quota of 50,000 microseconds of every 100,000 (which is half a CPU), and at most 20 processes. The second run, with no limits, shows max, meaning unlimited. These paths are for the modern cgroup v2; on older hosts the files have different names, but the idea is the same.

Compare with namespaces from How Docker Works: namespaces limit what a container can see, cgroups limit what it can use.

Memory: --memory and the OOM killer

--memory (short -m) sets a hard ceiling. Suffixes are b, k, m and g. If the container's processes need more than the limit, the kernel's OOM killer picks a process in the container and kills it with SIGKILL.

Here is a small Python program that keeps allocating 10 MB blocks and prints a line for each:

Python
import time
data = []
for i in range(1, 51):
    data.append(b"x" * (10 * 1024 * 1024))
    print("allocated", i * 10, "MB", flush=True)
    time.sleep(0.05)

Run it with a 100 MB limit (I mounted the script read-only and used --memory-swap equal to --memory, explained below):

CONSOLE
$ docker run -d --name tcd-oom --memory 100m --memory-swap 100m \
    -v "$PWD/oom.py:/oom.py:ro" python:3-slim python /oom.py
$ docker logs tcd-oom | tail -3
allocated 70 MB
allocated 80 MB
allocated 90 MB
$ docker inspect -f 'status={{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}}' tcd-oom
status=exited exit=137 oom=true

The program got as far as 90 MB, and then needed more than 100 MB in total with the interpreter's own memory. The kernel killed it. Three clues tell you this was the OOM killer and not a normal crash: the exit code is 137 (128 + 9, SIGKILL), OOMKilled is true, and the log simply stops with no error message, because the process had no chance to print anything.

An OOM-killed container often looks like a mysterious silent crash. Whenever a container exits with 137 and no message, check docker inspect -f '{{.State.OOMKilled}}' before debugging anything else. The fix is either to raise the limit or, better, to fix the memory hunger.

Swap

In an earlier experiment with only --memory 100m, a similar program survived allocations up to about 190 MB before dying. By default Docker lets a container use swap equal to its memory limit on top of the limit (a container started with --memory 64m reports MemorySwap=134217728 in docker inspect, double the memory). Swap hides memory problems and makes the app slow instead of dead. For predictable behavior, set --memory-swap equal to --memory, which disables swap for the container:

CONSOLE
$ docker run --rm --memory 100m --memory-swap 100m alpine:3.22 sh -c 'cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.swap.max'
104857600
0

Other memory flags: --memory-reservation is a soft limit that the kernel enforces only when the host is short of memory; --oom-kill-disable stops the OOM killer from acting, which is almost always a bad idea, because a container that cannot be killed can freeze the host.

There is also a minimum: Docker refuses limits below 6 MB.

CONSOLE
$ docker run --rm --memory 4m alpine:3.22 echo hi
docker: Error response from daemon: Minimum memory limit allowed is 6MB

free lies inside containers

Tools such as free and top read the host's numbers from /proc, not the cgroup:

CONSOLE
$ docker run --rm --memory 64m alpine:3.22 free -m | head -2
              total        used        free      shared  buff/cache   available
Mem:          11906         955        4502          41        6450       10699

Despite the 64 MB limit, free reports almost 12 GB. Programs that size their caches from /proc/meminfo can therefore think they have far more than they do. Modern runtimes (Java, Go, .NET, recent Node) read the cgroup file instead, but old software and some tuning scripts do not. When you see an application using "too much" memory in a container, check what it believes its limit is.

CPU: --cpus, shares and pinning

CPU is a compressible resource: when a container exceeds its share, it is slowed down, not killed. --cpus sets a quota as a decimal number of CPUs:

CONSOLE
$ docker run -d --name tcd-cpu --cpus 0.5 alpine:3.22 sh -c 'while :; do :; done'
$ docker run -d --name tcd-cpu2 alpine:3.22 sh -c 'while :; do :; done'
$ docker run -d --name tcd-cpu3 --cpus 0.25 --memory 64m alpine:3.22 sh -c 'while :; do :; done'
$ docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'
NAME        CPU %     MEM USAGE / LIMIT
tcd-cpu     55.89%    352KiB / 11.63GiB
tcd-cpu2    108.48%   368KiB / 11.63GiB
tcd-cpu3    28.27%    788KiB / 64MiB

Three endless loops, each wanting a full core. The one limited to 0.5 CPU was held to roughly 50% (the sample read 55.9%), the one at 0.25 to roughly 25% (28.3%), and the unlimited one used a full core (about 100%; docker stats percentages are per-CPU and the sample was a little above 100). Memory is shown against the limit: 64MiB where I set one, and the whole VM (11.63GiB) where I did not.

The same container sees all the host's CPUs even when throttled:

CONSOLE
$ docker run --rm --memory 64m --cpus 0.5 alpine:3.22 nproc
6

--cpus limits time, not the number of cores a program sees, so a runtime that starts one worker per visible core (thread pools, GOMAXPROCS, gunicorn --workers) may create far more workers than the quota can feed, which hurts latency. Modern runtimes read the quota, but check yours.

To restrict which cores a container may use, pin it:

CONSOLE
$ docker run --rm --cpuset-cpus 0 alpine:3.22 nproc
1

Finally, --cpu-shares (default 1024) is a relative weight that matters only when CPUs are contended: a container with 2048 gets twice the CPU time of a 1024 container during a fight, and nothing when the host is idle. Shares are good for "prefer the web server over the batch job", while --cpus is a hard cap.

Processes: --pids-limit

A fork bomb, a leaking worker pool or a buggy script that spawns children can exhaust the host's process table. --pids-limit caps the number of processes and threads inside the container:

CONSOLE
$ docker run --rm --pids-limit 20 alpine:3.22 sh -c 'for i in $(seq 1 40); do sleep 5 & done; kill $(jobs -p) 2>/dev/null'
sh: can't fork: Resource temporarily unavailable

The shell started background sleep processes until the limit of 20 was reached, and then the kernel refused to create more with EAGAIN, printed as "Resource temporarily unavailable". The container survived and so did the host. A limit of a few hundred is a sensible default for typical services; databases and thread-heavy runtimes need more, since every thread counts.

File descriptors and other limits: --ulimit

--ulimit sets classic per-process limits: open files (nofile), maximum file size (fsize), core dump size (core), number of processes (nproc) and so on, as soft:hard.

CONSOLE
$ docker run --rm --ulimit nofile=64:64 alpine:3.22 sh -c 'ulimit -n; ulimit -Hn'
64
64
$ docker run --rm alpine:3.22 sh -c 'ulimit -n'
1048576

On this machine the default is about a million open files per process, which is generous. A value that is too low can stop the container from starting at all: with --ulimit nofile=16:16, the container runtime failed during startup with too many open files. Servers that handle thousands of connections (nginx, databases, Elasticsearch-like systems) often need higher limits than a distribution default, and --ulimit is where you set them.

Changing limits on a running container

docker update changes memory, CPU, PIDs and restart policy without recreating the container:

CONSOLE
$ docker run -d --name tcd-up --memory 50m alpine:3.22 sleep 300
$ docker update --memory 200m --memory-swap 200m tcd-up
tcd-up
$ docker exec tcd-up cat /sys/fs/cgroup/memory.max
209715200
$ docker stats --no-stream --format '{{.Name}} {{.MemUsage}} {{.MemPerc}} pids={{.PIDs}}' tcd-up
tcd-up 692KiB / 200MiB 0.34% pids=1

Note that when you lower a memory limit below current usage, the kernel may kill the process. Also remember that docker update does not change the Compose file, so the next docker compose up goes back to what the file says.

Seeing what a container uses

  • docker stats is a live table (CPU %, memory, network and disk I/O, process count); --no-stream prints one sample.
  • docker inspect shows the configured limits: Memory=67108864 Swap=134217728 NanoCpus=500000000 PidsLimit=100 for a container started with --memory 64m --cpus 0.5 --pids-limit 100.
  • docker events --filter event=oom reports OOM kills as they happen.

In Compose, the same settings live under deploy.resources.limits (and reservations), or the older mem_limit and cpus keys on a service (Services in Compose).

Choosing sensible values

Resource Starting point Watch for
Memory Measure real use under load, then add 25-50% headroom OOMKilled, exit 137
Swap Same as memory (no swap) Surprising slowness when swap is allowed
CPU --cpus from measured use; leave unlimited for latency-sensitive services if the host has spare capacity Throttling, worker counts based on visible cores
PIDs 100-500 for typical services can't fork, thread-heavy runtimes
Open files Default is usually fine; raise for busy servers too many open files errors

Runtimes with their own memory management need their limits told about the container. The JVM, for example, sizes its heap from the cgroup limit by default on modern versions, but you should still set -XX:MaxRAMPercentage deliberately (Containerizing a Java App). For how to find the right numbers, see Debugging Containers.

Capacity planning: sizing a host

Limits protect one container from its neighbors. Capacity planning answers a different question: how many containers fit on one machine before something has to give? The tools are the same, but now you add the numbers up.

Limits, reservations and overcommit

  • A limit (--memory, --cpus) is a ceiling. The kernel enforces it with the OOM killer (memory) or throttling (CPU).
  • A reservation (--memory-reservation, --cpu-shares) is a soft preference about what the container should get when the host is under pressure. It is not a guarantee and it does not set any memory aside. On cgroup v2, --memory-reservation 128m becomes memory.low=134217728, and --cpu-shares 512 becomes cpu.weight=59.
  • Docker does not check that your limits add up. On a 12 GB test machine, three containers with --memory 8g all started, and so did one with --memory 20g. That is overcommit: the sum of the limits is larger than the host.

Overcommit is normal and useful, because services rarely peak at the same moment. It is only dangerous when limits are far above real use and the peaks do coincide. Then the host runs out of memory and the kernel kills something, and that "something" may be your database, not the container that caused the problem.

A worked example: a 16 GB, 4-core host

Say you run a web app, a worker, PostgreSQL and Redis. You measured each one under load with docker stats and found these peaks:

Service Peak memory Peak CPU Limit you set Reservation
web (x2) 350 MB 0.8 core 512 MB, 1 core 384 MB
worker (x2) 600 MB 1.0 core 768 MB, 1 core 512 MB
postgres 2.5 GB 1.5 cores 3 GB, 2 cores 3 GB
redis 400 MB 0.3 core 512 MB 512 MB

Add it up, and keep the host's own needs in the sum:

  • Limits: 2 x 512 MB + 2 x 768 MB + 3 GB + 512 MB = 6 GB. Reservations add up to about 5.3 GB.
  • Set aside about 2 GB for the OS, the Docker daemon, logging and monitoring agents, and keep another 20-25% of the machine free so the page cache has room.
  • Result: on a 16 GB host you use 6 GB of limits and 2 GB of system overhead, leaving about 8 GB free. There is room to add services, or you could pick a smaller host. A host where limits add up to more than roughly 70-80% of RAM is the one to worry about.

CPU is more forgiving, because the kernel time-slices it. The sum of peak CPU here is about 5.4 cores on a 4-core host, so the machine can be CPU-bound if everything peaks together. That is acceptable for a batch worker and not for a latency-sensitive web app, which is why the web containers should get a higher --cpu-shares than the workers.

Signals that a host is too small

Watch these over days, not minutes:

Signal Where to look What it tells you
Host memory above ~80% for long periods docker stats, host metrics (Metrics and Dashboards) Little headroom for the page cache or a spike
OOMKilled: true, exit code 137 docker inspect, docker events --filter event=oom A limit is too low, or there is a leak
CPU throttling (nr_throttled rising in cpu.stat) cAdvisor metric container_cpu_cfs_throttled_periods_total --cpus is too tight, even if average CPU looks low
Swap in use on the host free -m on the host or VM The overcommit is already hurting
Load average well above the core count host metrics CPU is saturated and latency will suffer
Disk above ~80% or high I/O wait docker system df, host metrics Images, logs and volumes need cleanup or a bigger disk

On Docker Desktop, the VM is the host. It has its own CPU and memory budget (on the test machine, docker info reported 6 CPUs and about 12.5 GB), which is smaller than your laptop. Size against that number, and see Docker Desktop and WSL2 to change it.

A simple routine

  1. Start with generous limits and measure peaks under a realistic load test.
  2. Set limits at peak plus 25-50%, and reservations near typical use.
  3. Add the numbers up as above, keeping 20-30% spare. If they do not fit, split services across hosts (Single Host Deployment, Swarm Basics).
  4. Re-check after every big release, because memory use grows quietly.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A container exits with code 137 and no error message in the logs. What should you check first?

    137 is 128 + 9 (SIGKILL). If State.OOMKilled is true, the kernel killed the process for exceeding the memory limit.
  2. 2. What does --cpus 0.5 do to a container that wants a full core?

    --cpus sets a CFS quota (50,000 out of every 100,000 microseconds). The container is throttled, not killed, and still sees all CPUs with nproc.
  3. 3. Why set --memory-swap equal to --memory?

    By default the container can use swap equal to its memory limit on top of it. Setting both equal leaves no swap.
  4. 4. Inside a container started with --memory 64m, free -m shows about 12 GB total. Why?

    Many tools read host-wide values from /proc. The real limit is in /sys/fs/cgroup/memory.max.
  5. 5. Which flag protects the host from a fork bomb inside a container?

    --pids-limit caps the number of processes and threads. When it is reached, fork fails with 'Resource temporarily unavailable'.

Summary

  • Limits are enforced by the kernel through cgroups; unlimited shows as max in /sys/fs/cgroup.
  • --memory is a hard cap. Exceeding it triggers the OOM killer: exit code 137 and OOMKilled=true. Use --memory-swap equal to --memory to disable swap. Docker refuses values under 6 MB.
  • --cpus throttles run time (CPU is compressible); --cpu-shares is a relative weight under contention; --cpuset-cpus pins cores. Containers still see all cores with nproc.
  • --pids-limit stops fork bombs and leaks; --ulimit sets per-process limits such as open files.
  • free and top show host numbers; check the cgroup files or docker stats for the truth.
  • docker update changes limits live; Compose has deploy.resources.limits.

Next: Signals, PID 1 and --init