Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 30 ยท Operations and Performance

Reloads, Zero-Downtime Deploys and Containers

Reload nginx and Apache without dropping requests, see how old workers drain, switch backends blue-green, run both servers in Docker and test configs in CI.

Sooner or later you must change a running web server: a new virtual host, a renewed certificate, a different backend. The naive way is restart, which stops the server and starts it again. For a moment nothing listens on port 443, in-flight downloads are cut off, and a visitor sees an error. On a busy site "a moment" is hundreds of failed requests.

Both nginx and Apache have a better answer, the graceful reload: start new workers with the new configuration, let the old workers finish what they are doing, then retire them. This lesson shows how that works, how to deploy new application versions without downtime, how to run both servers in containers, and how to test configuration in CI so a typo never reaches production.

Real commands and configuration appear in read-only blocks, because the site cannot run a web server or Docker. The interactive examples are JavaScript stand-ins: a simulation of workers draining during a reload, and a re-implementation of envsubst. They model the behavior, not the real servers.

Restart or reload?

Restart Stop everything, start again. Short outage.
Reload New workers start, old ones finish and leave.
Test first nginx -t or apachectl configtest before any reload.

Reload for config changes. Restart only when you must, such as new modules.

1 / 2

What a graceful reload does

In nginx, one master process owns the configuration and a set of worker processes serves traffic. When you reload, this sequence happens:

sequenceDiagram
    participant A as Admin
    participant M as Master process
    participant O as Old workers
    participant N as New workers
    A->>M: nginx -s reload (SIGHUP)
    M->>M: Read and test new config
    M->>N: Start with new config
    M->>O: Graceful shutdown
    O->>O: Stop accepting, finish open requests
    O-->>M: Exit when idle

The new workers accept every new connection from the moment they start; the old ones only finish what they already have.

Two details make this safe. If the new configuration has an error, the master logs it and keeps the old workers running, so a failed reload is a non-event. And because old and new workers overlap briefly, memory use briefly doubles and a very slow download can keep an old worker alive for a long time. For long-lived connections such as WebSockets, set a limit so old workers cannot stay forever:

NGINX
worker_shutdown_timeout 30s;     # after 30 s of draining, close what remains

The commands:

Terminal
sudo nginx -t                          # always first
sudo nginx -s reload                   # or: sudo systemctl reload nginx
sudo nginx -s quit                     # graceful stop (finish requests, then exit)
sudo nginx -s reopen                   # reopen log files after rotation

Under the hood these send signals to the master: HUP reloads, QUIT stops gracefully, TERM stops immediately, USR1 reopens logs. nginx can even upgrade its own binary without dropping connections (USR2, then QUIT the old master), though package managers usually do that for you.

Apache works similarly, with different names:

Terminal
sudo apachectl configtest              # Syntax OK?
sudo apachectl graceful                # or: sudo systemctl reload apache2
sudo apachectl graceful-stop           # finish requests, then stop
sudo apachectl restart                 # NOT graceful: children are told to stop at once

With graceful, each child finishes its current request and exits, and new children use the new configuration. Settings that need a full restart are the exception: loading a different MPM or a new module that changes how the server starts. If a module list change does not take effect after a reload, a restart is the safe step. Apache can be less forgiving than nginx about a broken file, so configtest before graceful is not optional.

Watch the draining happen

Here is a simulation of the difference. A request arrives every 200 ms. Most take 150 ms, but every seventh is a slow 1.5-second download. At t = 2000 ms the admin either restarts (the process dies and nothing listens for 800 ms) or reloads.

JS Stand-in: restart versus graceful reload

Press Run (or Ctrl + Enter) to see the console output here.

No output.

In the restart run, one slow download was cut off and four connections were refused. In the reload run, the slow download that started before the switch (connection 7) finished normally on generation 1, and everything after t = 2000 ms went to generation 2. The old workers disappeared only after their last connection ended. Real servers add details, such as connections queued by the kernel during the switch, but the shape is the same.

Zero-downtime application deploys

A web server reload covers configuration. Deploying new application code is a different question: how do you replace the backend that nginx or Apache proxies to without a gap? Three common techniques:

  1. Let the application server do it. PHP-FPM, Gunicorn, Puma and Node process managers can reload their workers gracefully, so the proxy never notices.
  2. Atomic directory switch. Deploy each release into its own directory and flip a current symlink in one step. For PHP-FPM, reload PHP-FPM afterward so its opcode cache does not serve old files, and keep open_file_cache_valid short (see Performance Tuning).
  3. Blue-green. Run the new version (green) next to the old one (blue), check it, then move traffic with one reload.

A blue-green switch in nginx can be as small as an included file:

NGINX
# /etc/nginx/upstreams/active.conf -> symlink to blue.conf or green.conf
upstream app {
    server 10.0.0.12:3000;        # green
}

# in the server block
location / {
    proxy_pass http://app;
}
Terminal
curl -fsS http://10.0.0.12:3000/healthz                        # 1. verify green directly
ln -sfn /etc/nginx/upstreams/green.conf /etc/nginx/upstreams/active.conf   # 2. flip
sudo nginx -t && sudo nginx -s reload                           # 3. graceful switch
# rollback = point the link back at blue.conf and reload again

Because a reload is graceful, requests already in flight finish on blue while new ones go to green. Keep blue running for a few minutes after the switch so rollback is instant. The traps are not in the web server: database migrations must work with both versions at once (add columns first, remove old ones in a later release), and user sessions must not live only in one version's memory. Apache can do the same with an Included file holding a ProxyPass or balancer definition plus apachectl graceful (balancers are covered in Load Balancing and Health Checks).

Running nginx and Apache in Docker

Official images exist for both, and they are the quickest way to try a configuration:

Terminal
# nginx: content in /usr/share/nginx/html, config in /etc/nginx
docker run -d --name web -p 8080:80 \
  -v "$PWD/site:/usr/share/nginx/html:ro" \
  -v "$PWD/nginx.conf:/etc/nginx/conf.d/default.conf:ro" \
  nginx:stable

# Apache httpd: content in /usr/local/apache2/htdocs, config in /usr/local/apache2/conf
docker run -d --name web2 -p 8081:80 \
  -v "$PWD/site:/usr/local/apache2/htdocs:ro" \
  httpd:2.4

Habits that carry over from this lesson:

Terminal
docker exec web nginx -t               # test the config inside the container
docker exec web nginx -s reload        # graceful reload, no container restart
docker exec web2 apachectl -t          # Apache syntax check
docker exec web2 apachectl graceful
docker logs -f web                     # logs go to stdout/stderr in the official images

Mount configuration read-only (:ro). A reload only rereads files, so edit the file on the host, test, and reload; the container keeps running. Since containers are disposable, treat the image plus mounted config as the deployable unit. For least privilege, prefer an unprivileged image variant (for example nginxinc/nginx-unprivileged, which listens on 8080) as discussed in Hardening and Security Headers.

Config templating with envsubst

Containers get their settings from environment variables, but nginx configuration files have no ${VAR} expansion of their own. The official nginx image bridges that gap: files named *.template in /etc/nginx/templates/ are run through envsubst at start-up and the results are written to /etc/nginx/conf.d/.

NGINX
# /etc/nginx/templates/default.conf.template
server {
    listen ${PORT};
    server_name ${DOMAIN};
    location / {
        proxy_pass http://${BACKEND};
        proxy_set_header Host $host;
    }
}

The catch is that nginx variables such as $host look exactly like shell variables. A naive substitution would blank them out. The official image restricts replacement to variables that exist in the container's environment, and outside Docker you can list the names yourself (envsubst '${PORT} ${DOMAIN} ${BACKEND}' < in > out). Try both behaviors:

JS Stand-in: envsubst and nginx $variables

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The naive run silently produced proxy_set_header Host ;, a configuration that is syntactically fine and functionally broken, which is exactly why you still test after templating. Apache's httpd image has no templating step, but Apache itself understands ${VAR} from the environment in its configuration (for example Listen ${PORT}), and it fails with a clear error if the variable is undefined.

Testing configuration in CI

A deploy should never be the first time a configuration is parsed. Both servers can check a file without starting for real:

Terminal
nginx -t -c "$PWD/nginx.conf"            # syntax and referenced files (needs the include paths to exist)
httpd -t -f "$PWD/httpd.conf"            # Apache syntax check

In a pipeline, run them inside the same image you deploy, so versions and modules match:

YAML
# .github/workflows/config.yml (excerpt)
- name: Test nginx config
  run: |
    docker run --rm \
      -v "$PWD/nginx/nginx.conf:/etc/nginx/nginx.conf:ro" \
      -v "$PWD/nginx/conf.d:/etc/nginx/conf.d:ro" \
      nginx:stable nginx -t

Common surprises: nginx -t also resolves host names in upstream and proxy_pass lines, so a backend name that exists only on your production network fails the test (host not found in upstream) unless you add a dummy entry, run the test on that network, or use a variable-based proxy_pass with a resolver. Certificate files named in ssl_certificate must exist too, so generate throwaway ones in CI. Beyond syntax, add a smoke test: start the container, curl a few URLs and assert on status codes and headers, as the capstone projects do (see Project: Host a Static Site with HTTPS and Caching).

A reload checklist that never fails you: edit, nginx -t (or apachectl configtest), reload, curl -I the site, glance at the error log. If the last step looks wrong, revert the file and reload again, because configuration in version control makes a rollback a one-command job.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What happens to a request that is already being served when you run nginx -s reload?

    The old workers stop accepting new connections but finish their current requests. New connections go to workers with the new configuration.
  2. 2. You reload nginx with a syntax error in the new configuration. What do you get?

    The master tests the new configuration first. If it fails, it keeps the working one. That is why you still run nginx -t beforehand to see the message immediately.
  3. 3. Why can naive envsubst break an nginx template?

    envsubst cannot tell shell-style variables from nginx variables. Restricting it to a list of names leaves $host and friends alone.
  4. 4. In a blue-green deploy, why keep the old (blue) backend running after switching?

    As long as blue is alive, a bad release is undone with one more graceful reload. Shutting blue down immediately removes that safety net.
  5. 5. Which command is NOT graceful?

    apachectl restart tells the children to stop at once, so in-flight requests can be dropped. graceful and reload let them finish.

Summary

  • A reload starts new workers with the new configuration and lets old workers drain; a failed reload leaves the old configuration serving.
  • Use nginx -t && nginx -s reload and apachectl configtest && apachectl graceful; avoid restart for routine changes. worker_shutdown_timeout caps draining for long-lived connections.
  • Zero-downtime application deploys use graceful app-server reloads, atomic symlink switches or blue-green upstream switching; keep the old version alive for rollback and keep migrations compatible.
  • In Docker, mount config read-only, test and reload with docker exec, and read logs with docker logs.
  • envsubst templates need an explicit variable list so nginx $variables survive.
  • Run nginx -t or httpd -t in CI inside the image you deploy, and add a curl smoke test.

Next: Project: Host a Static Site with HTTPS and Caching.