Skip to content
Tutorials Center
My learning

Topics

Lesson 44 of 111 ยท Containers in Depth

Docker Model Runner, Extensions and Docker Debug

Meet three Docker add-ons: Model Runner for local AI models and Compose models, Desktop Extensions, and docker debug for shells in slim or stopped containers.

Docker is more than run, build and compose. Around the engine sit tools that save you time once the basics are solid. Three of them come up often: Docker Model Runner, which runs AI models locally behind an OpenAI-compatible API; Docker Desktop Extensions, which add panels to the Desktop dashboard; and docker debug, which gives you a rich shell inside containers that have no shell at all.

This lesson explains what each one is, shows the commands, and says plainly where they stop. All three belong to Docker Desktop (Model Runner can also be installed on Docker Engine for Linux), so see Docker Engine, Desktop and licensing first if you are unsure which you have.

On the course machine (Docker Desktop, Docker 29.8.1, Compose v5.5.1) the command lists, help text and the Compose file below were checked. Actually pulling and running a model downloads gigabytes and needs a capable CPU or GPU, so the model examples are read-only: they are shown as the documented workflow, not as output we captured.

Docker Model Runner

A large language model (LLM) is a file of weights plus an engine that runs it. Running one on your own machine means choosing an engine (llama.cpp and others), finding a model file, managing GPU access and exposing an API. Model Runner wraps that in the Docker workflow:

  • Models are OCI artifacts. You pull them from Docker Hub (the ai/ namespace) or from Hugging Face, store them locally, and push your own, just like images.
  • The engine does not run inside a normal container. It runs on the host next to the daemon, so it can use the GPU directly (Apple silicon, NVIDIA and others, depending on platform). A model is loaded into memory when the first request arrives and unloaded when idle.
  • It serves an OpenAI-compatible HTTP API, so any client library that talks to OpenAI can talk to it by changing the base URL.

The commands

docker model --help lists the whole family. The everyday ones:

Terminal
docker model status                 # is the runner up?
docker model pull ai/smollm2        # download a small model
docker model list                   # models on this machine
docker model run ai/smollm2 "Say hello in five words"   # one prompt
docker model run ai/smollm2         # interactive chat
docker model ps                     # which models are loaded in memory
docker model unload ai/smollm2      # free the memory
docker model rm ai/smollm2          # delete from disk
docker model df                     # disk used by models

Other subcommands include inspect, search, tag, package and push (publish your own model), logs, requests (see recent calls), bench and gateway. On Docker Engine for Linux the runner is installed separately with docker model install-runner and controlled with start-runner, stop-runner and uninstall-runner.

Pick the model to match your memory: the file size is roughly the RAM it needs, plus room for the context. A "small" model of a few hundred megabytes runs on almost any laptop. Seven-billion-parameter models want 8 GB or more.

Calling the API

From inside a container, the runner is reachable at a special DNS name, and from the host through an optional TCP port:

Terminal
# from the host (after enabling host-side TCP support in Desktop settings)
curl http://localhost:12434/engines/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"ai/smollm2","messages":[{"role":"user","content":"Hello!"}]}'

# from a container
curl http://model-runner.docker.internal/engines/v1/chat/completions ...

The paths follow the OpenAI API (/chat/completions, /models, /embeddings), so existing code only needs a different base URL and a placeholder API key.

Models in Compose

Recent Compose versions can declare models next to services. Docker pulls the model, starts the runner if necessary, and injects the endpoint into the container as environment variables. This file was validated with docker compose config on Compose v5.5.1:

YAML
services:
  app:
    image: alpine:3
    models:
      llm:
        endpoint_var: AI_URL      # env var that will hold the API base URL
        model_var: AI_MODEL       # env var that will hold the model name

models:
  llm:
    model: ai/smollm2
    context_size: 4096

The app reads AI_URL and AI_MODEL from its environment and does not need to know where the runner lives. That keeps code identical between your laptop and a server with a different setup. See Services in Compose and Environment and Config for the surrounding file.

Limits to keep in mind

  • Local models are smaller and less capable than hosted frontier models. They are best for development, private data, offline work and cheap batch jobs.
  • A model in memory competes with your containers for RAM. Check Capacity Planning before running a large one next to a database.
  • Model files are large. They appear in docker model df, not in docker system df.
  • Treat prompts like any other input: models can be tricked by hostile text, so never let one run shell commands from untrusted content.

Docker Desktop Extensions

An extension is a Docker image with a special label and metadata file that Docker Desktop shows as a tab in its dashboard. It can ship a small web UI, a backend container, and host binaries (for example a CLI). Typical examples are log viewers, disk-usage explorers, database browsers and vendor tools.

The CLI has a few commands for them (docker extension --help on Desktop):

Terminal
docker extension ls                    # what is installed
docker extension install IMAGE:TAG     # install from an image
docker extension update IMAGE:TAG
docker extension rm IMAGE:TAG
docker extension init my-extension     # scaffold a new one from a template
docker extension validate IMAGE        # check the metadata before publishing

Extensions are convenient, but they run with your Desktop privileges and can include host binaries, so install only ones from publishers you trust, just as you would for any app. They are a Desktop feature: they do not exist on Docker Engine for Linux servers, and nothing in a production setup should depend on them. If a tool matters for your team's workflow, also keep a plain compose.yaml that runs it (Running Dev Tools in Containers).

docker debug

Minimal images (distroless, scratch, -slim variants) are great for security and size, and miserable for debugging: no shell, no ps, no curl. The traditional answer is a debug sidecar (Debugging Containers covers docker exec, docker cp, --pid and ephemeral debug images).

docker debug CONTAINER is a shortcut for that pattern. It attaches a toolbox shell to the target without changing the container or its image:

Terminal
docker debug my-container                 # shell in a running container
docker debug --preserve-user my-container # keep the container's user, not root
docker debug --privileged my-container    # extra capabilities, for deep digging
docker debug some-image:tag               # inspect an image or stopped container
docker debug -c 'ls /; ps' my-container   # run commands and exit

The options (--command, --host, --preserve-user, --privileged, --shell) are shown by docker debug --help. Inside the shell you get common tools (such as ps, curl, vim, htop), a view of the target's filesystem, and installed helpers, while the target's own files stay untouched. It works with stopped containers and with images too, which docker exec cannot do.

Some details to remember:

  • It ships with recent Docker Desktop versions as a CLI plugin. Which features are available can depend on your Docker plan, so check the current documentation.
  • The shell is root by default. --preserve-user is the safer choice when you want to see what the application itself can access.
  • Treat it like docker exec: it is a way into a production container, so restrict who can run it (Daemon Security).
  • Changes you make in the toolbox are not saved into the image. To fix the problem permanently, change the Dockerfile.

Choosing the right tool

Need Tool
Run an LLM locally with an OpenAI-style API Model Runner (docker model, Compose models:)
Add a GUI panel inside Docker Desktop An extension
Get a shell in a distroless or stopped container docker debug
A server or CI machine with no Desktop Plain Engine, Compose, and a debug sidecar

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How do you talk to a model served by Docker Model Runner from code?

    Model Runner exposes endpoints such as /engines/v1/chat/completions that follow the OpenAI API, so existing client libraries only need a different base URL.
  2. 2. What does the top-level models: element in a Compose file do?

    A service references a model with endpoint_var and model_var. Compose injects the API endpoint and the model name, so the app does not hard-code them.
  3. 3. Why is docker debug useful on a distroless image?

    docker debug attaches a shell with common tools to the target without modifying it, even if the target has no shell or is stopped.
  4. 4. Which statement about Docker Desktop Extensions is correct?

    Extensions are a Desktop feature. Because they can include backend containers and host binaries, trust matters, and production setups should not depend on them.

Summary

  • Model Runner pulls models as OCI artifacts, runs them on the host (using the GPU where available) and serves an OpenAI-compatible API. Use docker model pull|run|ps|rm, and declare models in Compose with the models: element.
  • Local models trade capability for privacy, cost and offline use, and they take real RAM and disk space.
  • Extensions add UI and tools to Docker Desktop; install only trusted ones and keep a plain Compose alternative for teams.
  • docker debug attaches a toolbox shell to running or stopped containers and images, including ones with no shell, without changing them.
  • None of these replace the fundamentals: images, containers, volumes, networks and Compose.

Next: Docker Metrics, Tracing and Dashboards.