Skip to content
Tutorials Center
My learning

Topics

Lesson 43 of 57 ยท Security, Monitoring and Cost

Azure Monitor, Log Analytics and Application Insights

Learn Azure Monitor: metrics, logs and traces, Log Analytics workspaces, KQL basics, alerts and action groups, Application Insights and availability tests, with a query stand-in.

You cannot fix what you cannot see. Once an application runs in the cloud, you no longer attach a debugger to it. You learn what it is doing from the data it emits: numbers about its health, text about what happened, and traces that follow a single request through many services. Azure Monitor is the umbrella service that collects, stores, queries and alerts on all of that data.

This lesson explains the three kinds of telemetry, how Log Analytics and KQL (Kusto Query Language) work, how alerts and action groups turn data into notifications, and what Application Insights adds for application code. The real queries are read-only because they need a workspace with data. The runnable examples are Python stand-ins: a tiny query engine that runs KQL-style steps over log rows, and an alert-rule evaluator.

Metrics, logs and traces

Monitoring data comes in three shapes, and each answers different questions:

  • Metrics are numbers sampled over time, such as CPU percentage, request count or queue length. They are small, cheap, stored for 93 days, and available within about a minute, which makes them ideal for dashboards and fast alerts. A metric tells you that something is wrong ("failures went up").
  • Logs are records with fields: an exception with its message, a request with its URL and status code, a sign-in event. They are larger and slower to query but rich. A log tells you what happened and to whom.
  • Traces link the work done for one request across several services. Each step is a span with a start time, duration and a shared operation ID. A trace tells you where the time went.

A rule of thumb: alert on metrics, investigate with logs, locate with traces.

flowchart LR
    A["App Service<br/>VMs, Functions, SQL"] --> M["Azure Monitor<br/>metrics store"]
    A --> L["Log Analytics<br/>workspace (logs)"]
    I["Application Insights<br/>SDK in your code"] --> L
    M --> AL["Alert rules"]
    L --> AL
    AL --> AG["Action group<br/>email, SMS, webhook, Function"]
    L --> D["Workbooks<br/>and dashboards"]
    M --> D

Resources send metrics and logs to Azure Monitor; alert rules watch them and trigger action groups.

Platform logs, diagnostic settings and workspaces

Most Azure resources produce platform metrics automatically, with nothing to configure. Their resource logs (a firewall rule matched, a blob was read, a Key Vault secret was accessed) are different: they are produced only when you create a diagnostic setting that sends them somewhere. The destinations are a Log Analytics workspace, a storage account for cheap archiving, or an event hub for streaming to other tools.

A Log Analytics workspace is the database for logs. It is a regional resource, billed mainly by the amount of data ingested and the days kept (a short default retention is free; longer retention costs more). Data lives in tables such as AzureActivity (control-plane operations), AppRequests and AppExceptions (application data), or StorageBlobLogs.

Terminal
# Create a workspace and send a Key Vault's audit logs to it (read-only examples)
az monitor log-analytics workspace create \
  --resource-group rg-demo --workspace-name log-demo --location westeurope

az monitor diagnostic-settings create \
  --name send-to-logs \
  --resource $(az keyvault show -n kv-demo-4821 --query id -o tsv) \
  --workspace log-demo \
  --logs '[{"category":"AuditEvent","enabled":true}]'

Two practical tips. First, use one workspace per environment or region as a starting point rather than one per resource, so queries can join related data. Second, logging costs money by volume, so send what you will use, and set a daily cap or an alert on ingestion (see Pricing, Cost Management and Budgets).

KQL basics

You query logs with Kusto Query Language. A query starts with a table name and then passes the rows through a chain of operators joined by the pipe character |, like a shell pipeline. Each operator receives a table and returns a table.

KUSTO
AppRequests
| where TimeGenerated > ago(1h)
| where Success == false
| summarize failures = count() by Name
| order by failures desc
| take 10

The operators you will use every day:

Operator What it does Example
where Keeps rows that match a condition where ResultCode >= 500
project Chooses (and renames) columns project Name, DurationMs
extend Adds a calculated column extend slow = DurationMs > 400
summarize Aggregates rows into groups summarize count(), avg(DurationMs) by Name
order by Sorts order by failures desc
take Returns N arbitrary rows take 10
top Sorts and takes N top 5 by DurationMs desc
bin() Rounds a value to a bucket, used for time series summarize count() by bin(TimeGenerated, 5m)
render Draws a chart in the portal render timechart

Two habits will save you time and money. Filter by time first: queries scan only the time range you ask for, so where TimeGenerated > ago(1h) makes them fast. And filter early, project early, before the expensive operators.

A very common shape is a time series of errors:

KUSTO
AppRequests
| where TimeGenerated > ago(24h)
| summarize requests = count(), failed = countif(Success == false) by bin(TimeGenerated, 15m)
| extend failureRate = round(100.0 * failed / requests, 1)
| render timechart

A KQL-style engine you can run

The Python program below is not KQL and not Azure. It is a stand-in, about 40 lines long, that implements the same ideas: a table is a list of rows, and each operator takes a table and returns a new table. Reading how summarize groups rows by key and applies aggregate functions shows what the real engine does at a very large scale. The sample requests rows are generated deterministically, with the /api/orders route failing more often late in the data, and the three queries are written next to their KQL equivalents.

Python A tiny KQL-like query engine

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output answers two questions an on-call engineer would ask. Which route is slow? /api/search, with an average around 570 ms against about 120 ms for the others. Which route is failing? /api/orders has most of the 500 errors. Two different problems hide in one table, and a simple grouping separated them. Try changing by=["name"] to by=["minute"] to see failures over time.

Alerts and action groups

An alert rule watches a signal and fires when a condition is met. There are several types:

  • Metric alerts evaluate a metric against a static threshold or a dynamic one learned from history (for example CPU above 80 percent averaged over 5 minutes).
  • Log search alerts run a KQL query on a schedule and fire when the result meets a condition (for example more than 10 rows).
  • Activity log alerts fire on control-plane events, such as someone deleting a resource group.
  • Service health alerts fire on Azure incidents that affect your regions and services.

An alert rule does not say who gets told. That belongs to an action group, a reusable list of actions: email, SMS, push, voice call, webhook, Azure Function, Logic App (see Logic Apps and API Management) or an ITSM ticket. One group can serve many rules, and one rule can use several groups.

Terminal
az monitor action-group create \
  --resource-group rg-demo --name ag-oncall --short-name oncall \
  --action email ops [email protected]

az monitor metrics alert create \
  --resource-group rg-demo --name high-cpu \
  --scopes $(az vm show -g rg-demo -n vm-web --query id -o tsv) \
  --condition "avg Percentage CPU > 80" \
  --window-size 5m --evaluation-frequency 1m \
  --severity 2 --action ag-oncall

Alerts have severity from 0 (critical) to 4 (verbose), and a state: fired until the condition clears, then resolved. A good alert is actionable: someone should be able to do something about it. Too many noisy alerts train people to ignore all of them.

The second stand-in evaluates a metric alert against a list of per-minute failure counts. The rule is "average failures over the last 3 minutes greater than 5", and it notifies an action group only when the state changes, not on every evaluation.

Python Evaluating a metric alert over a window

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The spike starts at minute 8, but the alert fires at minute 9: averaging over a window avoids reacting to a single bad minute, at the cost of a small delay. It resolves at minute 13, when the average falls back under the threshold. Choosing window size and threshold is the same trade-off you met with autoscale rules in Scaling and Autoscale.

Application Insights

Application Insights is the application performance monitoring (APM) feature of Azure Monitor. You add an SDK or an agent to your code (many App Service and Functions apps can enable it with a switch), and it collects, with almost no work, the things you would otherwise write by hand:

  • incoming requests with duration and result code;
  • outgoing dependencies: calls to databases, HTTP APIs, queues;
  • exceptions with stack traces;
  • traces (your log messages) and custom events and metrics;
  • the application map, a diagram of your services and the health of calls between them;
  • live metrics, a near real-time stream for watching a deployment;
  • distributed tracing, correlating one operation across services via the operation ID.

Modern Application Insights stores its data in a workspace-based resource, so you query it with the same KQL as other logs. The SDKs are moving towards the open standard OpenTelemetry, which keeps your instrumentation portable.

An availability test goes beyond collecting data: Azure probes your site from several locations around the world at a regular interval and raises an alert if a URL fails to respond or answers wrongly. It catches a problem that internal metrics miss, such as DNS, a certificate or a firewall rule that makes the site unreachable for outsiders.

Pair this with the health endpoint and probes from Load Balancer, Application Gateway and Front Door. For release monitoring, Application Insights data is what a canary comparison relies on (see Deployment Slots, Blue-Green and Canary).

Dashboards and workbooks

Data is only useful if people can see it. Azure offers several views:

  • Metrics Explorer for charting metrics ad hoc.
  • Dashboards in the portal: pinned tiles from metrics, queries and resources, shareable with a team.
  • Workbooks: interactive reports that mix text, parameters, KQL queries and charts, good for runbooks and review meetings.
  • Azure Managed Grafana, if your team already uses Grafana.

Start with one dashboard per application showing the four golden signals: latency, traffic, errors and saturation. If you can see those four, you can usually find the problem.

Monitoring virtual machines: VM insights and everyday queries

The basics above work for any resource. Virtual machines need a few extra pieces, because Azure can see a VM's host metrics (CPU, disk and network from the hypervisor) but not what happens inside the operating system (memory use, processes, event logs).

VM insights closes that gap. It installs the Azure Monitor Agent (AMA) as a VM extension, driven by a data collection rule (DCR) that states what to collect and which Log Analytics workspace receives it. You get ready-made performance charts and a map of the processes and connections of each VM. Under the hood the data is in tables you can query with KQL: Heartbeat (a signal about once a minute), Perf or InsightsMetrics (counters), Syslog (Linux logs) and Event (Windows event logs).

KUSTO
// Which machines have stopped reporting for more than 5 minutes?
Heartbeat
| summarize LastBeat = max(TimeGenerated) by Computer
| where LastBeat < ago(5m)

// Average CPU per machine in 5-minute buckets (Linux and Windows)
Perf
| where ObjectName == "Processor" and CounterName == "% Processor Time" and InstanceName == "_Total"
| summarize AvgCpu = avg(CounterValue) by Computer, bin(TimeGenerated, 5m)

// Free memory and disk, using the VM insights table
InsightsMetrics
| where Namespace == "Memory" and Name == "AvailableMB"
| summarize MinFreeMb = min(Val) by Computer

// Errors in the Linux system log, and Windows error events
Syslog   | where SeverityLevel in ("err", "crit", "alert", "emerg") | project TimeGenerated, Computer, ProcessName, SyslogMessage
Event    | where EventLevelName == "Error" | summarize count() by Computer, Source

Alerts on these queries use a log search alert rule (see the alert section above): the rule runs the KQL on a schedule and fires when the result meets a threshold, for example "any machine whose last heartbeat is older than 5 minutes". A missing heartbeat catches a dead VM, a broken agent and a network cut with one rule.

Retention also matters: a workspace keeps data for a default period (shorter on the free allowance), and you can extend it per workspace or table, at a cost per GB-month; for long-term, rarely queried logs use cheaper archive options.

Terminal
az monitor log-analytics workspace update --resource-group rg-ops \
  --workspace-name law-prod --retention-time 90

A workbook such as the built-in VM performance workbook combines several of these queries with parameters (a time range, a subscription, a VM picker) in one interactive report, which is a better review-meeting tool than a screenshot of a chart.

A heartbeat check: a stand-in

The stand-in implements the first two queries above over invented rows: it finds silent machines and averages CPU in 5-minute buckets.

JS Heartbeat and CPU queries over sample rows

Press Run (or Ctrl + Enter) to see the console output here.

No output.

A starting setup

  1. Create a workspace and enable Application Insights for each app.
  2. Turn on diagnostic settings for the resources that matter (Key Vault, SQL, storage, firewalls).
  3. Create one action group for the on-call team.
  4. Create a few alerts: availability, failure rate, response time, saturation, plus an activity log alert for deletes.
  5. Review costs and retention after a month.

More VM logs and queries: inventory, CPU ranking and what to collect

Choosing what the agent collects

The Azure Monitor Agent only sends what a data collection rule (DCR) tells it to. Beyond the basic performance counters, a DCR can add:

  • Windows performance counters (\Processor(_Total)\% Processor Time, \Memory\Available MBytes) and Windows event logs (choose the channel, such as System or Application, and severity levels).
  • Linux performance counters and Syslog. For Syslog you add a facility (auth, cron, daemon, kern, syslog, user...) and pick the minimum severity per facility, so you collect auth at Info for sign-in auditing but daemon only at Warning to save cost.
  • IIS logs in W3C format from a Windows web server, which land in the W3CIISLog table.

Every extra source costs ingestion money, so collect what you will query or alert on. A DCR is a resource: change it, and every associated VM changes at its next refresh. Set the workspace data retention (30 days is included by default) per table to keep audit logs longer and chatty tables shorter.

The VMComputer inventory table

With VM insights and its dependency feature enabled, the VMComputer table holds one record per machine: Computer, OperatingSystemFamily, OperatingSystemFullName, AzureSize, Ipv4Addresses, Ipv4SubnetMasks, Ipv4DefaultGateways, MacAddresses, AgentId and more. It is a quick inventory of operating systems, sizes and IP addresses across all machines in a workspace:

KUSTO
VMComputer
| where TimeGenerated > ago(1d)
| summarize arg_max(TimeGenerated, *) by Computer
| project Computer, OperatingSystemFullName, AzureSize, Ipv4Addresses, MacAddresses

The arg_max(TimeGenerated, *) by Computer line keeps only the newest record for each machine, which is the standard pattern for "current state" from a table that appends every few minutes.

Ranking CPU across a fleet

To rank VMs by CPU, including machines from several subscriptions that report to the same workspace, compute the average and maximum per machine and sort them:

KUSTO
Perf
| where TimeGenerated > ago(24h)
| where ObjectName == "Processor" and CounterName == "% Processor Time" and InstanceName == "_Total"
| summarize AvgCpu = avg(CounterValue), MaxCpu = max(CounterValue) by Computer
| order by AvgCpu desc

order by ... desc puts the busiest machines first (candidates to resize up); asc shows the idlest (candidates to shrink or switch off). To compare VMs from many subscriptions, query a workspace that all of them send to, or use the portal's cross-workspace query. The stand-in runs the same logic on invented samples.

JS Average and maximum CPU per computer

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Diagnostic settings categories and alert suppression

A subscription diagnostic setting exports the Activity Log to a workspace, a storage account or an event hub. Its categories include Administrative, Security, ServiceHealth, Alert, Recommendation, Policy, Autoscale and ResourceHealth. Keep Administrative, Security and Policy for audits (many compliance standards such as ISO 27001, HIPAA and GDPR expect an audit trail), and ServiceHealth and ResourceHealth to learn when Azure itself is having problems.

An alert rule can run actions (an action group) and can be suppressed: an alert processing rule mutes notifications during planned maintenance or a window you choose, without deleting the rule. Use it so a deliberate reboot at 02:00 does not page anyone. See Monitoring with Azure Monitor above for creating the rule itself.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which kind of telemetry is best for a fast alert such as 'CPU above 80 percent for 5 minutes'?

    Metrics are small numeric samples available within about a minute, which makes them the natural base for fast alerts. Logs are richer but slower and used to investigate.
  2. 2. What does the KQL query AppRequests | where Success == false | summarize count() by Name return?

    where keeps only failed rows, then summarize ... by Name groups them by name and counts each group.
  3. 3. Why does a resource's log data often not appear in Log Analytics until you configure something?

    Platform metrics flow automatically, but resource logs need a diagnostic setting that routes them to a workspace, storage account or event hub.
  4. 4. What is the job of an action group?

    Alert rules decide when to fire. Action groups are reusable lists of actions such as email, SMS, webhooks and Functions.
  5. 5. What does an Application Insights availability test do that internal metrics might miss?

    An external probe sees what users see, so it catches problems such as a broken certificate, DNS issue or firewall rule that make the site unreachable.

Summary

  • Azure Monitor collects metrics (fast numbers), logs (rich records) and traces (request paths across services). Alert on metrics, investigate with logs, locate with traces.
  • Resource logs need a diagnostic setting that sends them to a Log Analytics workspace, storage or an event hub; logging cost grows with volume.
  • KQL pipes a table through operators: where, project, extend, summarize, order by, take, bin, render. Filter by time first.
  • Alert rules watch signals; action groups decide who is told. Make alerts actionable and watch out for noise.
  • Application Insights adds requests, dependencies, exceptions, traces, the application map and availability tests to your code with little effort.
  • Start with a workspace, diagnostic settings, one action group, a handful of alerts and a golden-signals dashboard.

Next: Security Best Practices and Defender for Cloud.