Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 111 ยท Images

Layers and the Union Filesystem

See how Docker image layers work: union filesystems, copy-on-write, whiteout files, docker history, and why deleting a file in a later layer does not shrink an image.

Every image you pull or build is a stack of layers, and every container adds one more on top. That one design decision explains an astonishing number of Docker behaviors: why builds are fast the second time, why pulls download only part of an image, why a container's changes vanish when it is removed, and why deleting a huge file in a later RUN step does not make your image smaller.

In this lesson you will look at the layers of a real image, watch copy-on-write happen inside a container, open an image's archive to find the files in each layer (including the special marker Docker uses for deletions), and learn the habits that follow from all this.

What a layer is

A layer is a set of filesystem changes: files added, modified or deleted relative to the layer below. It is stored as a tar archive. The complete filesystem of a container is what you get by stacking the layers in order and merging them, so that for every path the topmost layer that mentions it wins.

flowchart BT
    L1["Base layer: debian:bookworm-slim"] --> L2["RUN apt-get install ..."]
    L2 --> L3["COPY requirements.txt + RUN pip install"]
    L3 --> L4["COPY . /app"]
    L4 --> C["Container: thin writable layer on top"]

An image is a stack of read-only layers, one per Dockerfile instruction. A running container adds one thin writable layer on top.

TEXT
  container  โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”  read-write (thin, per container)
             โ”‚ writable layer              โ”‚
  image      โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค  read-only
             โ”‚ RUN echo "config v2" ...    โ”‚  layer 4
             โ”‚ RUN rm /big.bin             โ”‚  layer 3
             โ”‚ RUN dd ... /big.bin         โ”‚  layer 2
             โ”‚ RUN echo "config v1" ...    โ”‚  layer 1
             โ”‚ alpine base (one layer)     โ”‚  layer 0
             โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
        merged view the process sees:  /, /etc, /bin, ...

The software that merges the layers into one directory tree is a union filesystem. On Linux Docker uses overlayfs (the kernel feature behind the classic overlay2 storage driver). Overlayfs takes a list of read-only "lower" directories and one writable "upper" directory and presents them as a single merged directory. Docker Desktop with Docker 29 reports it like this:

CONSOLE
$ docker info -f '{{.Driver}} {{.DriverStatus}}'
overlayfs [[driver-type io.containerd.snapshotter.v1]]

overlayfs through the containerd snapshotter is the current default for new installs; older installs show overlay2. For you as a user the behavior is identical: layers are read-only directories, and a container gets a writable one.

Looking at layers with docker history

We will build a deliberately wasteful image to study. It creates a 20 MB file and then deletes it, in separate steps:

DOCKERFILE
FROM alpine
RUN echo "config v1" > /etc/app.conf
RUN dd if=/dev/zero of=/big.bin bs=1M count=20 2>/dev/null
RUN rm /big.bin
RUN echo "config v2" > /etc/app.conf
CMD ["cat", "/etc/app.conf"]
CONSOLE
$ docker build -t tcd-layers:1 .
$ docker history tcd-layers:1
IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
cf3392ba7190   1 second ago    CMD ["cat" "/etc/app.conf"]                     0B        buildkit.dockerfile.v0
<missing>      1 second ago    RUN /bin/sh -c echo "config v2" > /etc/app.cโ€ฆ   12.3kB    buildkit.dockerfile.v0
<missing>      2 seconds ago   RUN /bin/sh -c rm /big.bin # buildkit           4.1kB     buildkit.dockerfile.v0
<missing>      2 seconds ago   RUN /bin/sh -c dd if=/dev/zero of=/big.bin bโ€ฆ   21MB      buildkit.dockerfile.v0
<missing>      3 seconds ago   RUN /bin/sh -c echo "config v1" > /etc/app.cโ€ฆ   12.3kB    buildkit.dockerfile.v0
<missing>      40 hours ago    CMD ["/bin/sh"]                                 0B        buildkit.dockerfile.v0
<missing>      40 hours ago    ADD alpine-minirootfs-3.24.3-x86_64.tar.gz /โ€ฆ   9.08MB    buildkit.dockerfile.v0

Each row is a build step; the newest is on top. Notice three things:

  • Instructions that only set metadata (CMD) have size 0B: they change the image config, not the filesystem. There are 5 filesystem layers here (docker image inspect -f '{{len .RootFS.Layers}}' tcd-layers:1 prints 5): the alpine base and our four RUN steps.
  • The dd step is 21 MB, which makes sense.
  • The rm /big.bin step is only 4.1 kB, not minus 21 MB. A layer can only add information.

Now the punchline. What does the finished image weigh?

CONSOLE
$ docker images tcd-layers
IMAGE             ID             DISK USAGE   CONTENT SIZE   EXTRA
tcd-layers:1      cf3392ba7190         34MB         3.88MB
$ docker run --rm tcd-layers:1 ls -la /big.bin
ls: /big.bin: No such file or directory

The file is gone for any process in a container (the ls fails), yet the image occupies 34 MB on disk: the alpine base (about 13 MB) plus the 21 MB layer that still holds big.bin. (The compressed "content size" is only 3.88 MB because a file of zeros compresses to almost nothing, but real files do not.) The data was never erased; a higher layer merely hides it. Anyone who pulls the image downloads it, and anyone who inspects the layers can still extract it. That includes a password you copied in and deleted later.

How deletion works: whiteouts

To prove it, we exported the image with docker save and listed what each layer archive contains (the archive is an OCI layout, blobs/sha256/<digest>):

CONSOLE
$ docker save tcd-layers:1 -o layers.tar && mkdir x && tar -xf layers.tar -C x
$ tar -tzvf x/blobs/sha256/50bbbbac...
-rw-r--r-- 0/0 20971520 big.bin
$ tar -tzvf x/blobs/sha256/e5fc6899...
---------- 0/0        0 .wh.big.bin

The first layer really contains the 20 MB big.bin (it compresses down to about 20 KB on the wire). The layer made by rm contains a zero-byte marker named .wh.big.bin. A file named .wh.<name> is a whiteout: it tells the union filesystem "treat <name> as deleted when merging". (Overlayfs itself uses a special device file for the same purpose; the .wh. form is the portable tar representation.) A deleted directory gets an "opaque" marker. Either way, the old layer is untouched, which is the rule: layers are immutable.

The fix: do it in one step

Create and remove the file in the same RUN, so that the intermediate state never gets committed to a layer:

DOCKERFILE
FROM alpine
RUN echo "config v1" > /etc/app.conf \
 && dd if=/dev/zero of=/big.bin bs=1M count=20 2>/dev/null \
 && rm /big.bin \
 && echo "config v2" > /etc/app.conf
CMD ["cat", "/etc/app.conf"]
CONSOLE
$ docker build -f Dockerfile.one -t tcd-layers:2 .
$ docker images tcd-layers
IMAGE             ID             DISK USAGE   CONTENT SIZE   EXTRA
tcd-layers:1      cf3392ba7190         34MB         3.88MB
tcd-layers:2      f5fb2e2d7fa6         13MB         3.86MB
$ docker image inspect -f '{{len .RootFS.Layers}}' tcd-layers:2
2

Same result, 13 MB instead of 34 MB, and 2 layers instead of 5. This is exactly why you will see long RUN apt-get update && apt-get install ... && rm -rf /var/lib/apt/lists/* chains in real Dockerfiles: the cleanup only saves space if it happens in the same layer as the download. FROM and RUN goes into that pattern, and multi-stage builds solve the same problem more elegantly by leaving build tools behind entirely.

Copy-on-write in a container

When you start a container, Docker adds a new writable layer (the overlayfs "upper" directory) on top of the image. The image layers stay read-only. How, then, can a process modify /etc/app.conf, a file that lives in a read-only layer?

By copy-on-write:

  • Reading a file serves it from the highest layer that has it. No copy.
  • Modifying an existing file first copies the whole file up into the writable layer ("copy-up"), then changes the copy. The original in the image layer stays untouched.
  • Creating a file just writes it into the writable layer.
  • Deleting a file writes a whiteout into the writable layer.

We can watch all four with two containers started from the same image:

CONSOLE
$ docker run -d --name c1 tcd-layers:2 sleep 300
$ docker run -d --name c2 tcd-layers:2 sleep 300
$ docker exec c1 sh -c 'echo "changed in c1" > /etc/app.conf; rm /etc/issue; echo new > /new.txt'
$ docker diff c1
A /new.txt
C /etc
C /etc/app.conf
D /etc/issue
$ docker exec c1 cat /etc/app.conf
changed in c1
$ docker exec c2 cat /etc/app.conf
config v2
$ docker run --rm tcd-layers:2
config v2

docker diff is literally a listing of the writable layer: Added /new.txt, Changed /etc/app.conf (copied up and modified), Deleted /etc/issue. The sibling container c2 and a brand-new container still see config v2: the image is shared and was never altered. When c1 is removed, its writable layer is deleted with it and the changes are gone.

Docker also bind-mounts a few files into every container (/etc/hostname, /etc/hosts, /etc/resolv.conf). They are managed by Docker and are not part of any layer, which is why rm /etc/hostname inside a container fails with "Resource busy".

Why writing heavily in the writable layer is slow

Copy-up and the extra indirection of a union filesystem cost something. A database or a busy log writer that works inside the container's own filesystem is slower than one writing to a volume, which bypasses the union filesystem and talks to the host directory directly. A large file that is modified in a tiny spot is copied up as a whole on first write. These are the practical reasons for the rule you will see throughout the course: anything that changes often or must outlive the container belongs in a volume.

Layers and the build cache

Docker records the result of every build step. When you build again, it checks each instruction from the top; if the instruction and its inputs have not changed, it reuses the stored layer instead of running the step. We changed nothing and rebuilt tcd-layers:1; BuildKit's plain progress output marks every step:

CONSOLE
$ docker build --progress=plain -t tcd-layers:1 .
#4 [1/5] FROM docker.io/library/alpine:latest@sha256:be2656061b53...
#5 [4/5] RUN rm /big.bin
#5 CACHED
#6 [2/5] RUN echo "config v1" > /etc/app.conf
#6 CACHED
#7 [3/5] RUN dd if=/dev/zero of=/big.bin bs=1M count=20 2>/dev/null
#7 CACHED
...

(Step numbers are not in order because BuildKit works in parallel; the bracketed [2/5] is the Dockerfile step.) Now we appended one new line at the end of the Dockerfile, RUN echo extra > /extra, and built again as tcd-layers:3: every earlier step still says CACHED, and only the new one runs. Had we edited an early line, that step and all steps after it would have been rebuilt, because each layer depends on the one below. This is why instruction order matters so much, and The Build Cache is dedicated to it.

Layers and pulling

The same sharing happens over the network. A registry stores layers by digest, and docker pull fetches only the ones you do not have (Already exists in the pull output). If you update the last line of your Dockerfile and push again, the registry receives, and your servers download, only that final layer. Putting rarely changing things (OS packages, dependencies) in lower layers and frequently changing things (your source code) in upper layers makes deployments small and fast.

Practical rules

  1. Clean up in the same layer that creates the mess (package caches, archives, build tools, temporary files).
  2. Never put secrets in a layer, not even temporarily. A later rm does not remove them from the earlier layer. Use BuildKit secret mounts instead.
  3. Order from stable to volatile: base image, system packages, dependency manifests and installs, then your source code.
  4. Fewer, purposeful layers, not "as few as possible". Merging unrelated steps wrecks the cache. Merge steps that belong together (install + cleanup).
  5. Use docker history to find what makes an image large; the SIZE column points at the guilty layer.
  6. Keep data out of the writable layer: volumes for state, docker diff to find out what a container writes.
  7. Remember the limits: union filesystems limit how many layers can be stacked (historically 127 for images), so extremely long chains of RUN steps eventually fail.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Your Dockerfile has 'RUN wget big-archive.tar.gz', then 'RUN tar xf big-archive.tar.gz', then 'RUN rm big-archive.tar.gz'. Why is the final image still large?

    Layers are immutable and only add information. The deletion is recorded as a whiteout in a higher layer, while the earlier layer still contains the data. Download, extract and delete in one RUN instead.
  2. 2. What is a whiteout file such as .wh.big.bin?

    Whiteouts represent deletions in a layer. The deleted file still exists physically in the lower layer.
  3. 3. A process in a container edits /etc/app.conf, which comes from an image layer. What happens?

    This is copy-on-write (copy-up). The original stays in the read-only layer, and the changed copy lives only in this container's writable layer.
  4. 4. What does the SIZE column of 'docker history' show for a CMD or ENV instruction?

    Metadata instructions (CMD, ENTRYPOINT, ENV, EXPOSE, LABEL, ...) add no filesystem layer and therefore have size 0B.
  5. 5. You change line 2 of a 10-line Dockerfile and rebuild. Which steps can use the build cache?

    Each layer depends on all layers below it, so after the first changed step the cache can no longer be trusted for the following steps.

Summary

  • An image is a stack of immutable layers; each layer is a tar of added, changed or deleted files relative to the layer below.
  • A union filesystem (overlayfs) merges the read-only layers plus one writable container layer into a single directory tree.
  • Containers use copy-on-write: reads come from the highest layer that has the file, writes copy the file up into the container's writable layer, and deletes create whiteouts (.wh.<name>).
  • Deleting a file in a later layer hides it but does not shrink the image; create and delete in the same RUN, or use multi-stage builds.
  • docker history shows the layer behind each build step and its size; metadata-only instructions are 0B.
  • Layers are shared between images and containers, cached during builds and downloaded once from registries, so order your Dockerfile from rarely changing to frequently changing.
  • Never put secrets into any layer, and keep changing data in volumes instead of the writable layer.

Next: Choosing a Base Image