Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 27 ยท Practice and Review

Interview Questions

Prepare for monitoring interviews with short model answers on Prometheus concepts, PromQL, alerting, architecture, scaling and Grafana, plus runnable stand-ins.

Interviewers for monitoring and SRE roles rarely want a recited definition. They want to hear that you understand why Prometheus works the way it does, what it costs, and where it breaks. This lesson collects the questions that come up most often, grouped by topic, with answers short enough to say out loud in about a minute. Each answer points back to the lesson that teaches the topic in depth.

Prometheus, Alertmanager and Grafana cannot run on this site, so the queries and configuration in the answers are real and read-only. The two TryIts are stand-ins in plain JavaScript that reproduce two points interviewers love to probe: why irate and rate disagree, and how fast label cardinality grows. They are models, not Prometheus.

What interviewers listen for

Reasoning Why, not only what.
Trade-offs Pull or push, rate or irate.
Experience Outages and fixes you have seen.
Numbers Cardinality, retention, scrape cost.

A short, honest answer beats a long, vague one.

1 / 2

Concepts

What is Prometheus, and how is it different from a logging system? Prometheus is a time series database with a pull-based collection model and a query language, PromQL. It stores numeric samples (a metric name plus labels, a timestamp and a value). Logs record individual events with text; metrics record aggregates that are cheap to store and fast to query. Metrics tell you that something is wrong and how bad it is; logs and traces help find why. See What Monitoring Is and Why It Matters.

Why does Prometheus pull instead of push? Pulling lets the server decide what to scrape and how often, makes "is the target alive?" a first-class signal (the up metric), and lets you open a target's /metrics page in a browser to debug it. Targets stay simple because they only expose current values. The cost is that short-lived jobs may finish before a scrape, and targets behind firewalls need to be reachable. See How Prometheus Works.

When is the Pushgateway the right tool? Only for service-level batch jobs that finish before they can be scraped, such as a nightly backup that reports when it last succeeded. It is the wrong tool for ordinary services: it never forgets a pushed series, hides target failures, and becomes a single point of failure. See Exporters: Node, Blackbox and Friends.

What are the four metric types? A counter only increases (and resets to zero on restart). A gauge goes up and down. A histogram counts observations into configurable buckets and exposes _bucket, _sum and _count. A summary computes quantiles on the client. You normally pick a histogram when percentiles must be aggregated across instances. See Counters, Gauges, Histograms and Summaries.

What is a time series, and what defines one? One unique combination of a metric name and a set of label key-value pairs. http_requests_total{method="GET", status="200"} and the same name with status="500" are two different series. Every label value you add can multiply the number of series, which is why cardinality matters. See Metrics, Labels and Time Series.

What does the up metric tell you? For every target, Prometheus generates up with value 1 if the last scrape succeeded and 0 if it failed, and it also records scrape_duration_seconds and scrape_samples_scraped. A failing up tells you the target, not the application's internal health, is unreachable. Application-level symptoms need their own metrics.

PromQL

Explain rate, irate and increase. rate(x[5m]) is the average per-second increase over the window, with counter resets handled. irate(x[5m]) uses only the last two samples in the window, so it is spiky and good for fast-moving graphs but poor for alerting. increase(x[5m]) is rate multiplied by the window length, so it answers "how many in the last five minutes". See rate, increase and Counter Math.

The stand-in below shows why they differ on the same data: a short burst makes irate jump while rate stays smooth.

JS rate versus irate on a bursty counter

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Why rate before sum? rate needs to see each individual counter to detect resets. If you sum first, one instance restarting looks like a drop in the total and the reset cannot be corrected. Write sum(rate(x[5m])), never rate(sum(x)[5m:]) for plain counters.

How do you compute a 95th percentile latency?

PROMQL
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Aggregate the bucket rates while keeping le, then let histogram_quantile interpolate. The result is an estimate: accuracy depends on bucket boundaries, and a quantile that falls in the last finite bucket is capped at its upper bound. See Quantiles, Histograms and SLI Queries.

Why can you not average percentiles? The average of two p95 values is not the p95 of the combined traffic. That is why histograms, whose buckets can be summed, beat client-side summary quantiles for services with many instances.

What do on, ignoring and group_left do? They control vector matching in binary operations. By default, both sides must have identical label sets. on(a, b) matches only on the listed labels, ignoring(c) matches on everything except them, and group_left allows several series on the left side to match one on the right (many-to-one), typically to copy extra labels such as version from an info metric. See Aggregation, Joins and Vector Matching.

What is staleness? When a series disappears from a target's output (or the target vanishes), Prometheus writes a stale marker so the series stops appearing in instant queries right away, instead of lingering for the default 5-minute lookback. It is why graphs end cleanly when a pod is replaced.

Alerting

What should you alert on? Symptoms that users or the business feel (high error ratio, high latency, an SLO burning down), not every possible cause (high CPU, a single failed disk). Causes belong on dashboards. Every page should be urgent, actionable and need a human. See Alerting Rules and Alertmanager.

What does the for clause do? It requires the condition to stay true across evaluations for that long before the alert moves from pending to firing. It filters brief blips. If the condition becomes false while pending, the timer resets.

How does Alertmanager reduce noise? It groups related alerts into one notification, inhibits lower-priority alerts when a higher one is firing (for example, node-level alerts while the whole cluster is down), supports silences for planned work, and routes by label to different receivers. Prometheus decides when an alert exists; Alertmanager decides who is told, and how.

What is a burn-rate alert? An alert on how fast the error budget is being used. With a 99.9% objective, the budget is 0.1% errors. A burn rate of 14.4 means errors arrive at 14.4 times the sustainable pace, which would use 2% of a 30-day budget in one hour. Teams combine a long and a short window so alerts fire quickly but also stop quickly. See Project: Alerts with Burn Rates and Routing.

How do you test alert rules? With promtool check rules for syntax and promtool test rules for behavior: you supply synthetic input series and assert which alerts exist at which time.

Architecture and scaling

What limits a single Prometheus server? Mostly the number of active series (memory, since the current block lives in the head) and the ingestion rate, then the cost of slow queries and local disk. One well-sized server handles millions of series, but there is no built-in clustering. See Scaling, Retention and Long-Term Storage.

How do you make it highly available? Run two identical servers scraping the same targets, with the same rules, both sending alerts to a clustered Alertmanager that deduplicates them. Their data differs slightly, so dashboards pick one. This protects availability, not long-term history.

How do you keep data for years? Send samples with remote_write to long-term storage such as Thanos, Mimir or VictoriaMetrics, which can also downsample and query across many servers. Local retention (15 days by default) stays short on purpose.

What are recording rules for? They precompute expensive or frequently used expressions and store them as new series, so dashboards and alerts read a cheap series instead of recomputing. Convention for names is level:metric:operations, for example job:http_requests:rate5m. See Recording Rules and Query Performance.

What is the biggest operational risk, and how do you prevent it? A cardinality explosion: a label with unbounded values (user IDs, full URLs, request IDs) creates millions of series and exhausts memory. Prevent it in code review, with sample_limit on scrape jobs, and by watching prometheus_tsdb_head_series. See Security, Cardinality and Common Pitfalls.

The stand-in below makes the arithmetic visible: series count is the product of the label value counts.

JS How labels multiply series

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Grafana

What is the difference between a data source and a dashboard? A data source is a connection (Prometheus, Loki, a SQL database). A dashboard is a set of panels, each with queries against one or more data sources. Variables let one dashboard serve many services. See Grafana: Data Sources and Dashboards.

How do you manage dashboards as code? Provision data sources and dashboard JSON from files at start-up, keep them in Git and review changes like code, so a rebuilt Grafana looks exactly the same. See Designing Dashboards That Help.

What makes a dashboard useful in an incident? It starts with the symptoms (rate, errors, duration), puts the most important panels on top, uses correct units and thresholds, and links down to detail. A wall of 60 graphs that nobody can interpret is a design failure.

How to answer well

Say the trade-off first, then your pick. "Histograms, because I need to aggregate percentiles across instances; I would pay for it with bucket planning." Admit limits. And tie answers to something you measured or an incident you handled, even a small one.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which statement about irate is correct?

    irate uses only the two most recent samples, so it is spiky. rate averages across the window and suits alerting.
  2. 2. Why is a user_id label dangerous?

    Series count is the product of label value counts. Unbounded values lead to a cardinality explosion.
  3. 3. What should a paging alert normally measure?

    Alerts should be urgent and actionable. Symptoms that users feel qualify; causes usually belong on dashboards.
  4. 4. How do you get long-term history with Prometheus?

    Local storage is meant to be short. Remote storage adds long retention, downsampling and a global view.

Summary

  • Explain Prometheus as pull-based metrics with labels, a query language and local storage, and name when the Pushgateway is acceptable.
  • Know the rules behind rate before sum, histogram quantile accuracy, vector matching and staleness.
  • Alert on symptoms with for, and let Alertmanager group, inhibit, silence and route.
  • Scale with sensible limits, HA pairs, recording rules and remote storage, and guard against cardinality explosions.
  • Answer with trade-offs and real numbers.

Next, keep the essentials at hand in the Prometheus and Grafana Cheat Sheet.