Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 30 ยท Operations and Performance

Logs and Troubleshooting

Read nginx and Apache access and error logs, customize log formats, diagnose 403, 404, 502, 503 and 504, and debug with nginx -T, httpd -S and curl.

When a web server misbehaves, it is usually quite willing to tell you why. The explanation is sitting in a log file, and the skill that separates a calm fix from an hour of guessing is knowing which log to read, what a line means, and which command shows what the server actually loaded. Most outages are one of a short list of causes, and each leaves a recognizable trace.

This lesson walks through both kinds of log, shows how to change their format, maps the common error codes to their usual causes, and finishes with a toolbox of commands (nginx -T, apachectl -S, curl -I) that answer "what is this server really doing?".

Real server logs and commands appear in read-only blocks, because the site cannot run a web server. The interactive examples are JavaScript stand-ins: a log analyzer and an error-line explainer that process sample text. They show the technique, not a real server.

Two logs, two questions

Access log What was asked, and what did we answer?
Error log Why did something go wrong inside?

Start with the access log for the status code, then the error log for the reason.

1 / 2

Where the logs are

nginx Apache
Access log /var/log/nginx/access.log /var/log/apache2/access.log (Debian), /var/log/httpd/access_log (RHEL)
Error log /var/log/nginx/error.log /var/log/apache2/error.log, /var/log/httpd/error_log
Set with access_log, error_log CustomLog, ErrorLog

Follow a log live with sudo tail -f /var/log/nginx/error.log, and read the service's own messages (config errors at start-up, crashes) with sudo journalctl -u nginx -e (or -u apache2, -u httpd). Virtual hosts can have their own files, so check the access_log or CustomLog line of the site you are debugging, not only the default.

The access log

The access log has one line per request. nginx's default combined format looks like this:

Text
203.0.113.5 - - [06/Nov/2026:10:00:01 +0000] "GET /app.css HTTP/1.1" 200 8300 "https://example.com/" "Mozilla/5.0"

In order: client address, an unused identity field, the logged-in user, the time, the request line, the status code, the body size in bytes, the referrer and the user agent. Apache's combined format is the same layout.

The useful part is that you can change it. A format that adds timing is the first improvement almost everyone makes:

NGINX
http {
    log_format timed '$remote_addr - $remote_user [$time_local] "$request" '
                     '$status $body_bytes_sent "$http_referer" "$http_user_agent" '
                     'rt=$request_time urt="$upstream_response_time"';
    access_log /var/log/nginx/access.log timed;
}
APACHE
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" rt=%{ms}T" timed
CustomLog ${APACHE_LOG_DIR}/access.log timed

$request_time is the whole request in seconds; $upstream_response_time is how long the backend took, so a big gap between them points at slow clients or buffering, not at your application. Apache's %{ms}T logs milliseconds (%D is microseconds, and %T whole seconds). Also useful: nginx $request_id or Apache %{UNIQUE_ID}e (with mod_unique_id) give every request an ID you can also pass to the backend, so one line of the web log matches one line of the application log.

You can quiet noisy traffic too. Health checks from a load balancer fill the log with nothing of interest:

NGINX
location = /healthz {
    access_log off;
    return 200 "ok\n";
}

Analyze a log

Even without tools like GoAccess, a short script answers the main questions: how many requests per status class, which requests failed, which are slow. This analyzer reads sample lines in the timed format above.

JS Stand-in: summarize an access log

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Two things stand out. The 504 on /api/report took exactly 60 seconds (the default proxy read timeout), while the successful call took 2.4 s, so that endpoint is slow, not broken. And /missing.png is requested repeatedly: a broken link on your own page, easy to fix. On a real server the same questions are one-liners such as awk '{print $9}' access.log | sort | uniq -c | sort -rn, which counts status codes in the default format.

The error log and its levels

The error log is for the server's own account of problems: failed backend connections, permission errors, rewrite trouble, config warnings. Its verbosity is a level:

NGINX
error_log /var/log/nginx/error.log warn;     # debug, info, notice, warn, error, crit, alert, emerg
APACHE
LogLevel warn
# raise one module only, instead of the whole server:
LogLevel warn rewrite:trace3 ssl:info

Every level includes the more serious ones below it. The nginx default is error, Apache's is warn. For a problem you are chasing, turn the level up for a short time, reproduce it, and turn it back down: debug in nginx needs a binary built with debug support and produces a huge file, and Apache's trace levels per module (such as rewrite:trace3) are the cleanest way to see why a rewrite did not match.

After you rotate logs, tell the server to reopen its files (nginx -s reopen, or a graceful Apache reload), otherwise it keeps writing to the old, renamed file. Package installs include a logrotate rule that does this for you.

What the status codes usually mean

Read the status in the access log first, then look for the error log line with the same time.

Code Usual causes
403 Forbidden The worker user cannot read the file or enter a parent directory; no index file and listings are off; an allow/deny or Require rule; SELinux.
404 Not Found Wrong root or alias, a file not deployed, a try_files or rewrite that points to the wrong place, case-sensitive file names.
413 Content Too Large Upload larger than client_max_body_size (nginx) or LimitRequestBody (Apache).
502 Bad Gateway The backend refused the connection, crashed, or sent something invalid. Typical: PHP-FPM or the app is not running, wrong port or socket path.
503 Service Unavailable Intentional limits (limit_req returns 503 unless you changed it), every upstream server is marked down, or maintenance mode.
504 Gateway Timeout The backend accepted the request but did not answer within the proxy timeout. Slow query or slow code.

The pair people confuse most is 502 versus 504. A 502 is usually fast: nothing is there to talk to. A 504 is slow: something is there but too busy or stuck. The error log distinguishes them clearly (Connection refused versus timed out). Here is a small helper that applies those rules to sample lines:

JS Stand-in: explain an error-log line

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The last line has no rule on purpose: a warn about a buffered body is information, not an error. A table of known messages helps beginners, but the habit that matters is reading the actual line, especially the part in parentheses (13: Permission denied, 111: Connection refused), which is the operating system's own explanation.

Permission and SELinux problems

A 403 on a file that exists is very often permissions. A file is only readable if every directory above it has the execute bit for the worker user, which is easy to miss when the site lives in a home directory:

Terminal
namei -l /var/www/site/index.html     # shows owner and mode of each path element
ps -o user= -C nginx | sort -u        # which users run the workers

On RHEL-family systems (and others that enable it), SELinux adds a second layer. Files must carry a label the web server is allowed to read, and the server is not allowed to open arbitrary network connections unless you permit it. Typical symptoms: a 403 or Permission denied although classic permissions look right, or a 502 with (13: Permission denied) while connecting to upstream.

Terminal
getenforce                                            # Enforcing, Permissive or Disabled
ls -Z /srv/site                                       # labels; web content should be httpd_sys_content_t
sudo semanage fcontext -a -t httpd_sys_content_t "/srv/site(/.*)?"
sudo restorecon -Rv /srv/site                         # apply the label
sudo setsebool -P httpd_can_network_connect 1         # allow proxying to backends
sudo ausearch -m avc -ts recent                       # show recent denials

Do not "fix" it with setenforce 0 and forget it; use denials to find the right label or boolean.

See what the server really loaded

Many "my change does nothing" problems are really "the server is not using the file I edited". Ask the server itself:

Terminal
sudo nginx -t                  # test the syntax and referenced files
sudo nginx -T | less           # test and print the entire merged configuration, with every include
sudo apachectl configtest      # Syntax OK?
sudo apachectl -S              # how virtual hosts, ports and the default site were parsed
sudo apachectl -M              # loaded modules

nginx -T is the best single command here: it shows the final, complete configuration, so a stray server block in conf.d/ or a leftover default site shows up immediately. apachectl -S does the same job for virtual hosts, including which one answers requests that match nothing (see Server Blocks and Choosing a Server and Apache Virtual Hosts).

Reproduce it with curl

Browsers cache redirects, reuse connections and hide details. curl shows the raw exchange:

Terminal
curl -I https://example.com/                       # headers only (HEAD request)
curl -sv https://example.com/ -o /dev/null         # full request and response headers, TLS details
curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" https://example.com/
curl -I -H "Host: example.com" http://127.0.0.1/   # talk to one machine, pretend to be a site
curl --resolve example.com:443:203.0.113.10 -I https://example.com/   # test a new server before DNS moves
curl -IL http://example.com/                       # follow redirects and show every hop

The Host and --resolve tricks are how you test a virtual host before DNS points at it, and -IL is how you find a redirect loop. A related rule: when a browser shows the wrong thing but curl shows the right one, suspect the browser cache, a cached 301, or a CDN in front.

When something breaks, resist changing three things at once. Reproduce with curl, find the matching error log line, form one hypothesis, change one setting, run the config test, reload, and curl again.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A request returns 504 after exactly 60 seconds. Which is the most likely cause?

    504 means a timeout while waiting for the backend. A missing backend would give a fast 502 with a connection refused message instead.
  2. 2. Which nginx command prints the full effective configuration including all included files?

    -T tests the configuration like -t and also dumps it, so you can see every include and spot stray server blocks.
  3. 3. Why does a file with mode 644 still return 403 when it lives under /home/alice/site?

    The worker must be able to traverse every directory in the path. A home directory with mode 700 blocks it before the file is even considered.
  4. 4. What does $upstream_response_time tell you compared with $request_time?

    request_time covers the whole request including sending the response to the client, while upstream_response_time measures only the backend.
  5. 5. On a RHEL server a proxied app returns 502 and the error log says (13: Permission denied) while connecting to upstream. What should you check?

    SELinux can forbid the web server to open network connections. Check the denials with ausearch and enable the right boolean instead of disabling SELinux.

Summary

  • The access log shows one line per request with its status; the error log explains what went wrong inside. Match them by time.
  • log_format (nginx) and LogFormat (Apache) let you add timing, upstream time and request IDs; use access_log off for noisy health checks.
  • Error log levels run from debug to emerg; raise them briefly, and in Apache raise a single module with LogLevel warn rewrite:trace3.
  • 403 is permissions, index or rules; 404 is paths; 413 is size limits; 502 is a dead or invalid backend; 503 is limits or no live upstream; 504 is a slow backend.
  • Check directory traversal with namei -l and, on RHEL-like systems, SELinux labels and booleans.
  • nginx -T, apachectl -S and apachectl -M show what is really loaded; curl -I, -v, -H Host: and --resolve reproduce problems without a browser.

Next: Performance Tuning.