Skip to content
Tutorials Center
My learning

Topics

Lesson 44 of 52 ยท Testing, Performance and Deployment

Observability in Production

See inside a running ASP.NET Core app: custom metrics with Meter, distributed traces with ActivitySource, structured logs and correlation, with OpenTelemetry.

When something is slow at 3 a.m., you cannot attach a debugger. All you have is what the app reported about itself. Observability is the practice of making an app report enough that you can answer questions you did not think of in advance: why did latency double after the deploy, which downstream call is failing, what happened to this one user's request?

The industry calls the three kinds of signals logs, metrics and traces. .NET has first-class support for all three in the base library, and the same APIs feed every monitoring product, so you instrument once and choose a backend later. This lesson builds each signal by hand so you can see exactly what is produced, then shows how OpenTelemetry ships them out.

Three signals, three questions

flowchart TD
    A["Your app"] --> L["Logs<br/>What happened?<br/>discrete events"]
    A --> M["Metrics<br/>How much, how often?<br/>numbers over time"]
    A --> T["Traces<br/>Where did the time go?<br/>one request across services"]
    L --> B["Backend<br/>Seq, Loki, Grafana,<br/>Azure Monitor, Datadog"]
    M --> B
    T --> B

Logs describe events, metrics aggregate numbers, traces follow one request through many operations.

  • Logs are timestamped records of events. Good for details and debugging a single incident.
  • Metrics are cheap numeric aggregates (counters, gauges, histograms). Good for dashboards and alerts, because their cost does not grow with traffic.
  • Traces record a tree of timed operations (spans) for one request, even across services. Good for finding which step is slow.

You need all three. An alert fires from a metric ("error rate above 2%"), a trace shows which dependency is slow, and the logs for that trace explain why.

Metrics with System.Diagnostics.Metrics

The Meter type creates instruments. A Counter<T> only goes up (requests served), an UpDownCounter<T> goes both ways (items in a queue), a Histogram<T> records a distribution (request durations), and an ObservableGauge<T> is read on demand (current memory). Every measurement can carry tags, which become dimensions in the dashboard.

The instrument code never knows who is listening. If nothing listens, recording is nearly free. Collectors such as OpenTelemetry attach through MeterListener, and we can do the same to see the raw measurements:

C# Counters, histograms and a MeterListener

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Notice how the tags split one counter into several series (channel=web, channel=mobile). Use tags for low-cardinality values like status class, route template or region. Never tag with unbounded values such as user IDs or full URLs: every distinct combination becomes a separate time series, and a backend can be overwhelmed ("cardinality explosion").

ASP.NET Core and the runtime already publish many meters of their own: Microsoft.AspNetCore.Hosting (request counts and durations), Microsoft.AspNetCore.Server.Kestrel (connections), Microsoft.AspNetCore.RateLimiting, System.Net.Http and more. You get request metrics without writing any code; your own meters add business numbers on top.

In larger apps register a meter through dependency injection with IMeterFactory (services.AddMetrics() is on by default in ASP.NET Core). It ties the meter's lifetime to the container and keeps tests isolated, but a static Meter as above is perfectly fine for a small service.

Tracing with ActivitySource

.NET names its trace spans activities. An ActivitySource creates them, and Activity.Current flows automatically across await calls, so a child activity started inside a parent becomes its child with no plumbing. Each trace has a 16-byte TraceId shared by every span, and each span has its own SpanId and a ParentId.

As with metrics, nothing happens until a listener subscribes. StartActivity returns null when nobody listens, which is why you always write activity?.SetTag(...).

C# Nested activities form a trace

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

ReserveStock and ChargeCard are children of Checkout, because Activity.Current was Checkout when they started. A trace viewer draws this as a waterfall and shows at a glance that, say, ChargeCard took 800 ms. Setting the status to Error marks the span red in the viewer.

Crossing process boundaries

A trace becomes truly useful when it spans services. The glue is the W3C traceparent header, which looks like 00-<trace-id>-<span-id>-01. HttpClient adds it to outgoing requests when an activity is current, and ASP.NET Core reads it from incoming requests, so a single trace id follows the call chain automatically as long as a listener (usually OpenTelemetry) is registered in each service.

Structured logs and correlation

Logging and Diagnostics covered ILogger. The two habits that matter in production are:

  1. Use message templates, not string interpolation. logger.LogInformation("Order {OrderId} paid", id) keeps OrderId as a field a backend can filter on. $"Order {id} paid" produces only text.
  2. Attach the trace id to every log line. With Activity.Current set, the logging pipeline can include TraceId and SpanId automatically (the built-in scopes do this when ActivityTrackingOptions is enabled). Then "show me all logs for this slow trace" is one query.

The next example puts all three together in a real Kestrel app. A small middleware starts an activity (adopting the caller's traceparent), counts and times the request with a meter, and opens a logging scope carrying the trace id. A tiny capture logger lets us see the structured fields:

C# Metrics, traces and logs correlated in one request

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Read the output from the top. The trace id the server returned is the one the caller sent, so both services' spans join one trace. Requests that arrive without a traceparent header start a trace of their own (shown as <new-trace>). The counter has one series per status code. Each request-handling log line carries TraceId plus its own structured fields (Path, StatusCode). One id now connects the metric spike, the span and the log lines.

OpenTelemetry in production

Everything above is the producer side. OpenTelemetry (OTel) is the vendor-neutral standard and set of NuGet packages that collects these signals and exports them. A typical setup (read-only, because it needs packages) is:

C#
builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService("shop-api"))
    .WithMetrics(m => m
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddMeter("Shop.Orders"))                // your own meter
    .WithTracing(t => t
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSource("Shop.Checkout"))             // your own ActivitySource
    .UseOtlpExporter();                          // OTLP to a collector or backend

The exporter reads OTEL_EXPORTER_OTLP_ENDPOINT from the environment, so the same image runs in every environment. Logs flow through builder.Logging.AddOpenTelemetry(...) and gain the trace id automatically. A common deployment sends everything to an OpenTelemetry Collector, which forwards to Grafana, Jaeger, Azure Monitor, Datadog or another backend. Because your own code uses only Meter, ActivitySource and ILogger, switching backends never touches it.

For local development, .NET Aspire's dashboard and the standalone Aspire dashboard container show logs, metrics and traces from OTLP with no setup beyond one environment variable.

Practical advice

  • Instrument the boundaries: incoming requests, outgoing HTTP and database calls, queue consumers. The framework already does most of these.
  • Measure what the business cares about as well: orders placed, payments failed, jobs queued.
  • Mind cardinality in tags and in log fields used for aggregation.
  • Sample traces in high-traffic services (for example 5%), but keep metrics complete.
  • Never log secrets or personal data. Logs are copied to many systems and kept for months.
  • Alert on symptoms (error rate, latency percentiles), not on every cause; use traces and logs to investigate.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which signal is cheapest to keep at very high traffic and best suited to alerting on error rate?

    Metrics are pre-aggregated numbers, so their storage cost does not grow with each request. Logs and traces grow with traffic.
  2. 2. Why does source.StartActivity(...) sometimes return null?

    Without a listener (such as OpenTelemetry) no activity is created. That is why you use activity?.SetTag(...).
  3. 3. What is a bad choice for a metric tag?

    A user id has unbounded distinct values, so every user creates a new time series and causes a cardinality explosion.
  4. 4. How does one trace id follow a request from service A to service B?

    The traceparent header carries trace id and parent span id; the receiving service starts its activity as a child.
  5. 5. Why prefer logger.LogInformation("Order {OrderId} paid", id) over string interpolation?

    Message templates preserve named values as fields, while an interpolated string is just opaque text.

Summary

  • Observability means logs (events), metrics (numbers over time) and traces (one request across operations).
  • Meter and its Counter, Histogram and ObservableGauge record metrics; tags add dimensions, but must stay low-cardinality.
  • ActivitySource and Activity build traces; children are linked automatically through Activity.Current, and the traceparent header links services.
  • Use message templates for logs and include the trace id so logs, metrics and traces can be correlated.
  • Producers use only built-in APIs; OpenTelemetry collects and exports to any backend, configured through the environment.
  • Collect what the framework gives you for free, add business metrics, sample traces and keep secrets out of telemetry.

Next: Project: Todo API