Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 38 ยท Testing and Deployment

Telemetry, Logging and Observability

See what your Phoenix app is doing: telemetry events and metrics, LiveDashboard, Logger metadata, custom events with :telemetry.execute and error tracking.

Once an app is deployed, you need to answer questions fast. Why is this page slow? How many database queries per request? Did the error rate jump after the last release? Phoenix, Ecto and LiveView all emit telemetry events: tiny named signals with measurements ("this query took 1.2 ms"). You attach handlers to them, turn them into metrics, and send those to a dashboard.

The Erlang :telemetry library is not runnable here, so the Elixir code is read-only. The editors rebuild a miniature telemetry system and a metrics reporter in JavaScript.

What an event looks like

An event has three parts: a name (a list of atoms), measurements (numbers) and metadata (any context).

ELIXIR
:telemetry.execute(
  [:my_app, :orders, :placed],       # name
  %{total_cents: 4_250, items: 3},   # measurements
  %{currency: "USD", user_id: 7}     # metadata
)

Phoenix and its libraries already emit many events, for example:

Event Fired when Useful measurement
[:phoenix, :endpoint, :stop] a request finishes duration
[:phoenix, :router_dispatch, :stop] a route finished duration, route in metadata
[:my_app, :repo, :query] an Ecto query ran total_time, queue_time
[:phoenix, :live_view, :mount, :stop] LiveView mounted duration
[:phoenix, :channel_joined] a channel join finished duration

Durations are in native time units; convert with System.convert_time_unit/3 or let the metrics layer do it.

Telemetry.Metrics and the telemetry module

New Phoenix projects include lib/my_app_web/telemetry.ex. It lists metrics, declarations of how to summarize events, and starts a supervisor that polls VM statistics:

ELIXIR
def metrics do
  [
    summary("phoenix.endpoint.stop.duration", unit: {:native, :millisecond}),
    summary("phoenix.router_dispatch.stop.duration",
      tags: [:route], unit: {:native, :millisecond}),
    summary("my_app.repo.query.total_time", unit: {:native, :millisecond}),
    counter("my_app.orders.placed.total_cents"),
    last_value("vm.memory.total", unit: {:byte, :kilobyte}),
    last_value("vm.total_run_queue_lengths.total")
  ]
end

The metric types are:

  • counter: how many times an event fired.
  • sum: running total of a measurement.
  • last_value: the most recent value, good for gauges like memory.
  • summary: statistics (average, max, percentiles) over time.
  • distribution: counts per bucket, which time-series databases turn into percentiles.

Metrics are only descriptions. A reporter (LiveDashboard, StatsD, Prometheus, OpenTelemetry) reads them and does something with the numbers.

LiveDashboard

Phoenix.LiveDashboard is a page, built with LiveView, that shows the metrics above live, plus system information: processes, ETS tables, Ecto stats, request logging and more. It is mounted in the router:

ELIXIR
import Phoenix.LiveDashboard.Router

scope "/" do
  pipe_through [:browser, :require_admin]
  live_dashboard "/dashboard", metrics: MyAppWeb.Telemetry
end

In the generated project it is only enabled in development. In production protect it with authentication (see authorization), because it exposes internals.

Handling an event yourself

You can attach a function to any event, for instance to log slow queries:

ELIXIR
:telemetry.attach(
  "log-slow-queries",
  [:my_app, :repo, :query],
  &MyApp.Telemetry.handle_query/4,
  nil
)

def handle_query(_event, measurements, metadata, _config) do
  ms = System.convert_time_unit(measurements.total_time, :native, :millisecond)
  if ms > 100, do: Logger.warning("slow query (#{ms} ms): #{metadata.query}")
end

Handlers run in the process that emitted the event, so keep them fast and never let them crash (a failing handler is detached automatically). Do heavy work by sending a message to another process.

Logging with context

Logger is the other half. Add metadata so each line can be traced to one request. Phoenix generates a request_id for you (via Plug.RequestId); include it in the log format:

ELIXIR
# config/config.exs
config :logger, :default_formatter,
  format: "$time $metadata[$level] $message\n",
  metadata: [:request_id, :user_id]
ELIXIR
# in a plug or LiveView mount
Logger.metadata(user_id: user.id)
Logger.info("order placed", order_id: order.id)

Use levels with intent: debug for development detail, info for business events, warning for recoverable oddities, error for failures that need a human. Never log passwords, tokens or full card numbers (see web security).

Errors and tracing

Telemetry shows trends; you also need individual failures. Error trackers such as Sentry or Honeybadger have Elixir clients that hook Plug and Logger to capture exceptions with the request and the stack trace. OpenTelemetry adds distributed traces: one request followed through your endpoint, Ecto queries and outgoing HTTP calls, each with a duration. Add them to your dependencies only once you have real traffic to watch.

Runnable models

First, a tiny :telemetry: handlers are attached by name, and execute calls every handler for that event, wrapped so a crashing handler cannot break the caller.

Node.js A miniature :telemetry (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Second, the metric types. A reporter subscribes to events and folds each measurement into a counter, a sum, a last value and a summary.

Node.js counter, sum, last_value and summary (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The outlier of 220 ms barely moves the average but shows up in the maximum and the 95th percentile, which is why dashboards chart percentiles rather than averages.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What are the three parts of a telemetry event?

    A name list identifies the event, measurements hold numbers, and metadata holds context.
  2. 2. Which metric type is best for the current memory usage of the VM?

    A gauge-like value is represented by last_value: the most recent reading.
  3. 3. Why should telemetry handlers be fast?

    Handlers execute synchronously in the emitting process, so slow work should be handed to another process.
  4. 4. Why chart the 95th percentile instead of only the average?

    Outliers affect real users but barely change the mean.

Summary

  • Telemetry events carry a name, measurements and metadata; Phoenix, Ecto and LiveView emit many by default.
  • Telemetry.Metrics declares counters, sums, last values, summaries and distributions; reporters consume them.
  • LiveDashboard shows metrics and system internals live; protect it in production.
  • Handlers run in the emitting process, so keep them short and safe.
  • Add Logger metadata (request id, user id) and an error tracker; add tracing when traffic justifies it.

Next: Project: A Live Poll