Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 38 ยท Testing and Deployment

Observability and Production Hardening

Make an Axum service production ready with request IDs, structured tracing, metrics, body limits, CORS and rate limiting, with runnable Rust stand-ins for the key algorithms.

When something goes wrong at 3 a.m., you have only what your service recorded. Observability is the practice of making a running system explain itself: logs that tie to one request, metrics that show trends, and traces that show where time went. Hardening is the other half: limits and guards so one bad client cannot hurt everyone else.

Axum does this through Tower layers, so each concern is one .layer(...) call. The real crates cannot run on this site, so the code is read-only and the algorithms behind it are modeled with runnable stand-ins.

Structured logs and request IDs

Use the tracing crate and tower_http::trace::TraceLayer. It opens a span for each request, so every log line inside a handler carries the method, path and request ID automatically.

Rust
use axum::{http::Request, Router};
use tower_http::{
    request_id::{MakeRequestUuid, PropagateRequestIdLayer, SetRequestIdLayer},
    trace::TraceLayer,
};
use tracing_subscriber::{fmt, EnvFilter};

fn init_tracing() {
    tracing_subscriber::fmt()
        .with_env_filter(EnvFilter::from_default_env()) // RUST_LOG=info,my_service=debug
        .json() // one JSON object per line: easy for log platforms
        .init();
}

fn with_observability(router: Router) -> Router {
    let x_request_id = axum::http::HeaderName::from_static("x-request-id");
    router
        .layer(TraceLayer::new_for_http().make_span_with(|req: &Request<_>| {
            let id = req.headers().get("x-request-id")
                .and_then(|v| v.to_str().ok()).unwrap_or("-");
            tracing::info_span!("http", method = %req.method(), uri = %req.uri(), request_id = %id)
        }))
        .layer(PropagateRequestIdLayer::new(x_request_id.clone()))
        .layer(SetRequestIdLayer::new(x_request_id, MakeRequestUuid))
}

Layer order matters: the last .layer added runs first on the way in, so SetRequestIdLayer assigns the ID before the trace span reads it. Return the ID to the client in a response header; when a user reports a problem, ask for that ID and search your logs for it.

Log at the right level: error for things that need a human, warn for odd but handled, info for lifecycle events, debug for detail. Do not log passwords, tokens or full request bodies.

Metrics

Logs answer "what happened to this request?". Metrics answer "how is the system doing?": requests per second, error rate, latency percentiles. A common choice is Prometheus. Your service exposes a /metrics text page and a scraper reads it periodically.

The four numbers to start with, often called the golden signals, are latency, traffic, errors and saturation (how full the service is).

Stand-in: counters in Prometheus text format

Rust Request counters rendered as Prometheus text (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

Label by the route pattern (/todos/{id}), never by the raw path. A label per user ID or per URL creates millions of series and can bring your metrics system down.

Hardening: limits and guards

Rust
use axum::extract::DefaultBodyLimit;
use std::time::Duration;
use tower_http::{cors::CorsLayer, limit::RequestBodyLimitLayer, timeout::TimeoutLayer, compression::CompressionLayer};

fn hardened(router: Router) -> Router {
    router
        .layer(DefaultBodyLimit::disable())
        .layer(RequestBodyLimitLayer::new(1024 * 1024)) // 1 MiB
        .layer(TimeoutLayer::new(Duration::from_secs(10)))
        .layer(CompressionLayer::new())
        .layer(CorsLayer::new().allow_origin(["https://app.example.com".parse().unwrap()]))
}
  • Body size limit: without one, a client can stream gigabytes. Axum has a 2 MB default for its body extractors; set your own deliberately.
  • Timeouts: a slow client must not hold a connection forever.
  • CORS: browsers block cross-origin calls unless your server says they are allowed. Allow named origins, not *, when cookies or credentials are involved.
  • Compression saves bandwidth for JSON and text.
  • Security headers such as Strict-Transport-Security can be added with SetResponseHeaderLayer.

Rate limiting with a token bucket

Rate limiting protects login endpoints and expensive routes. The classic algorithm is the token bucket: a bucket holds up to N tokens and refills at a steady rate; each request takes one token; an empty bucket means answer 429 Too Many Requests. Crates such as tower_governor implement this per client IP. The stand-in below uses a simulated clock so its output is exactly reproducible.

Rust A token bucket with a simulated clock (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

The burst allowance lets normal clients send a few quick requests, while the refill rate caps the sustained rate. Include a Retry-After header in 429 responses.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the purpose of a request ID?

    With an ID in logs and in a response header, you can find exactly what happened to a reported request.
  2. 2. Why label metrics by route pattern rather than the full path?

    Labels such as /users/48151 make one series per user; /users/{id} makes one.
  3. 3. In a token bucket, what happens when the bucket is empty?

    An empty bucket means the client exceeded its allowance; it must wait for the refill.
  4. 4. Why set a request body size limit?

    Limits protect memory and bandwidth from abusive or buggy clients.

Summary

  • Use tracing with TraceLayer and a request ID so logs are structured and searchable.
  • Expose metrics such as request counts and latency; label by route pattern.
  • Layer body limits, timeouts, compression and CORS onto the router deliberately.
  • Rate limit sensitive routes with a token bucket and answer 429 with Retry-After.
  • Never log secrets or whole request bodies.

Next: Project: A URL Shortener