Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 27 ยท Grafana

Grafana: Data Sources and Dashboards

Install Grafana, connect it to Prometheus, build panels with PromQL queries, add variables and time ranges, and provision everything from files.

Prometheus has a web page for running queries, and it is great for investigating, but it is not a place to live. Grafana is the usual front end: it connects to Prometheus (and to dozens of other data sources), runs queries on your behalf and draws the results as graphs, gauges, tables and heatmaps, arranged in dashboards that a whole team can share.

Grafana does not store your metrics. It stores dashboards, users and data source settings, and asks the data source for numbers every time a panel is drawn. This lesson walks through installing Grafana, adding Prometheus, building a panel, using variables and time ranges, organizing dashboards in folders, and loading everything from files with provisioning. Design advice follows in Designing Dashboards That Help.

Grafana is a server with a graphical interface and cannot run on this site, so installation commands, screens and configuration are described or shown read-only. The runnable examples are JavaScript stand-ins for two ideas Grafana relies on: drawing a series as a chart, and substituting variables into a query. They are simplified models, not Grafana's code.

Grafana in four words

Data source Where numbers come from.
Query What to ask the data source.
Panel One visualization of a query.
Dashboard A page of panels.
1 / 2

Installing and signing in

Grafana is a single service. The quickest way to try it is Docker:

Terminal
docker run -d --name grafana -p 3000:3000 grafana/grafana-oss

Open http://localhost:3000. A fresh installation starts with the user admin and password admin, and asks you to choose a new password at first sign-in. On Linux servers you can instead install the grafana package from Grafana's repository and run it as a systemd service; in Kubernetes it is usually deployed with a Helm chart. In every case, pin a version and keep Grafana's own data (the SQLite file under /var/lib/grafana by default) on a persistent volume, or you will lose your dashboards when the container is replaced.

For anything beyond a laptop, put Grafana behind HTTPS, change the default admin password at once and prefer single sign-on or at least named user accounts. Settings live in grafana.ini and can be overridden with environment variables such as GF_SERVER_ROOT_URL (the pattern is GF_<SECTION>_<KEY>).

Adding Prometheus as a data source

In the UI, open Connections โ†’ Data sources โ†’ Add data source โ†’ Prometheus, and set the URL of your Prometheus server, for example http://prometheus:9090. Choose Server access (called proxy in files), so the Grafana server makes the requests and the browser never needs a route to Prometheus. Press Save & test; a green message means Grafana could query the server.

Two settings are worth knowing. The scrape interval field tells Grafana how often Prometheus scrapes, so it can pick sensible resolutions and step sizes (it should match your scrape_interval). And the default editor setting chooses between the visual query builder and the plain code editor.

Panels and queries

A dashboard is a grid of panels. To make a panel, add a visualization, choose the data source and write a query. In the code editor you type plain PromQL. For example, a panel showing the request rate per route:

PROMQL
sum by (route) (rate(http_requests_total{job="web"}[$__rate_interval]))

$__rate_interval is a built-in variable: Grafana fills it in with a window that is a safe multiple of the scrape interval and fits the current zoom level, so rate() always has enough samples. Prefer it to hard-coded windows like [5m] in dashboards; a window shorter than about four scrape intervals gives gaps.

Under the query, the Legend option decides how each series is named. {{route}} uses the value of that label, which is far more readable than the full label set. In the panel's options you choose the visualization:

  • Time series for values over time (the default and the most common);
  • Stat and Gauge for a single current number, with colors by threshold;
  • Table for lists, such as the top ten routes by error count;
  • Bar gauge, Heatmap (good for histogram buckets) and State timeline for specific cases.

You can place several queries in one panel (for example, requests and errors) and Grafana can also perform simple transformations on the results, such as joining tables or computing a new column, without changing the PromQL.

Time ranges and refresh

The picker at the top right sets the time range (last 6 hours, last 7 days, or an absolute window) and the auto-refresh interval. Dragging across a graph zooms into that range for the whole dashboard. Grafana picks a step so each panel receives about as many points as the panel has pixels, which explains why a seven-day graph looks smoother than any single scrape. A long range with a short window can be heavy on Prometheus; this is where recording rules pay off.

Set auto-refresh no faster than your scrape interval. A dashboard refreshing every 5 seconds against 30-second scrapes does ten times the work to show the same data.

What the panel really does

Behind each panel Grafana calls the Prometheus HTTP API query_range with a query, a start, an end and a step, and gets back series of timestamped values. Drawing is then plain graphics work. The stand-in below takes a series and renders it as a text chart, which is the same job a Time series panel does with pixels:

JS Turning a series into a text chart

Press Run (or Ctrl + Enter) to see the console output here.

No output.

A real panel adds axes, smoothing, one line per series, thresholds and hover tooltips, but the core is the same: map values to positions.

Variables

Variables turn one dashboard into many. A Query variable named job can be defined with the Prometheus function:

TEXT
label_values(http_requests_total, job)

Grafana runs it, shows the result as a dropdown at the top of the dashboard, and replaces $job in every panel query before sending it. Variables can allow multi-select and All; in that case use the regex matcher so the query still works: http_requests_total{job=~"$job"}. Variables can also depend on each other (choose a cluster, then the instance list only offers that cluster's instances), and a Custom or Interval variable covers fixed lists.

The substitution is textual, which explains most variable bugs: the string is inserted before PromQL is parsed, so quoting and regexes matter.

JS How variables expand into a query

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice that job=~"$job" becomes job=~"web|api", a valid regex matcher, which is why the =~ form is the right one for multi-select variables. Grafana's own formatting of multi-values is richer than this, but this is the idea.

Folders and organizing dashboards

Dashboards live in folders, which are also the unit of permissions: you can give a team edit rights to its own folder and read-only access elsewhere. Tags and the search box help find things later. A sensible start is one folder per team or service area, and a naming style such as Service / Overview and Service / Details. Dashboards can be exported as JSON for review, copying or storage in Git.

Provisioning: dashboards from files

Clicking everything together does not scale: nobody can review a click, and a replaced container loses the work. Provisioning makes Grafana read YAML files at startup (and re-read them periodically) from /etc/grafana/provisioning. A data source file:

YAML
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: 15s

And a dashboard provider, which tells Grafana to load every dashboard JSON file from a folder on disk into a Grafana folder:

YAML
apiVersion: 1
providers:
  - name: service-dashboards
    folder: Services
    type: file
    disableDeletion: true
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards

With allowUiUpdates: false, edits in the UI cannot be saved over the provisioned version, which keeps Git the source of truth. A typical workflow is to build a dashboard in the UI, export its JSON, commit it, and let provisioning load it in every environment. The next lesson takes this further under "dashboards as code".

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where does Grafana get the numbers it draws?

    Grafana stores dashboards and settings, not the metrics. Each panel sends its query to the data source.
  2. 2. Why is $__rate_interval preferred over a fixed [5m] in dashboard queries?

    A fixed window can be too short at some zoom levels, which causes gaps, or needlessly long at others.
  3. 3. A dashboard variable allows multiple selections. Which matcher should the query use?

    Multiple values expand to something like web|api, which only works as a regular expression, so use =~.
  4. 4. What is the main benefit of provisioning dashboards and data sources from files?

    Provisioned files can live in Git and be loaded in every environment, so the setup survives rebuilds.

Summary

  • Grafana draws data from data sources such as Prometheus; it stores dashboards, users and settings, not your metrics.
  • Add Prometheus with server (proxy) access and press Save & test; keep Grafana's data on a persistent volume.
  • A panel is a query plus a visualization; use $__rate_interval, legends like {{route}}, and thresholds for stat and gauge panels.
  • Variables are substituted into queries as text before PromQL is parsed; use =~ with multi-select.
  • Organize dashboards in folders and provision data sources and dashboards from files so they live in Git.

Next: Designing Dashboards That Help