Skip to content
Tutorials Center
My learning

Topics

Lesson 60 of 70 ยท Machine learning and IoT

AWS IoT Core and the IoT Family

Learn AWS IoT Core: the device registry, certificates and policies, MQTT topics, device shadows, the rules engine, Greengrass at the edge, and a typical sensor ingestion architecture.

A sensor in a greenhouse, a smart meter, a delivery van, a factory machine: each is a small device that produces readings and sometimes needs commands. Thousands or millions of them cannot each have a username and password in a database, and most have very little memory and an unreliable connection. AWS IoT Core is the managed service that solves the connection side: it authenticates devices, carries their messages, and routes the data to the rest of AWS.

This lesson explains the pieces of IoT Core, a typical data path, and the neighbors that appear in the "IoT family": Greengrass for the edge, SiteWise for industrial data, and the services you use to store and analyze readings.

IoT services cannot run on this site. The CLI and policy examples are real and read-only. The Python programs are stand-ins that re-implement two rules of IoT Core in plain Python: MQTT topic matching and a tiny rules-engine query.

The core pieces

Things and the registry. A thing is the record of one device in the device registry: a name, a type and optional attributes (for example site=greenhouse-3). Things can be organized into groups, so you can send a command or an update to every device in a group.

Certificates and policies. Each device normally holds an X.509 certificate and its private key, and proves its identity during the TLS handshake. A certificate alone grants nothing. You attach an IoT policy to it that lists exactly which actions the device may perform and on which resources. This is where you enforce least privilege, as in IAM policies, but IoT policies have their own format. A device that publishes only to its own topics looks like this (read-only):

JSON
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:Connect",
      "Resource": "arn:aws:iot:eu-west-1:123456789012:client/${iot:Connection.Thing.ThingName}"
    },
    {
      "Effect": "Allow",
      "Action": "iot:Publish",
      "Resource": "arn:aws:iot:eu-west-1:123456789012:topic/sensors/${iot:Connection.Thing.ThingName}/data"
    },
    {
      "Effect": "Allow",
      "Action": ["iot:Subscribe", "iot:Receive"],
      "Resource": [
        "arn:aws:iot:eu-west-1:123456789012:topicfilter/commands/${iot:Connection.Thing.ThingName}",
        "arn:aws:iot:eu-west-1:123456789012:topic/commands/${iot:Connection.Thing.ThingName}"
      ]
    }
  ]
}

The policy variable ${iot:Connection.Thing.ThingName} means one policy works for every device, and no device can publish as another. Registering a device with the CLI is two steps (read-only):

Terminal
aws iot create-thing --thing-name greenhouse-3-sensor-12
aws iot create-keys-and-certificate --set-as-active \
  --certificate-pem-outfile device.crt --private-key-outfile device.key

The message broker. Devices talk to IoT Core through a message broker. The main protocol is MQTT, a lightweight publish/subscribe protocol made for constrained devices, normally over TLS on port 8883. HTTPS and MQTT over WebSockets are also supported. Your account has its own endpoint, such as abc123-ats.iot.eu-west-1.amazonaws.com.

MQTT topics

In MQTT, messages are published to a topic, a path-like string such as sensors/greenhouse-3/temperature. Devices and applications subscribe to a topic filter, and the broker delivers every matching message. Two wildcards are allowed in filters:

  • + matches exactly one level: sensors/+/temperature matches sensors/greenhouse-3/temperature.
  • # matches everything from here down and must be last: sensors/# matches all topics that start with sensors/.

MQTT also has three quality-of-service levels. QoS 0 means at most once, QoS 1 means at least once. IoT Core supports those two. Choose QoS 1 when a lost message matters, and handle duplicates in your code.

Python MQTT topic filters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Device shadows

Devices are often offline. A device shadow is a JSON document stored in IoT Core that holds a device's reported state (what the device says it is) and desired state (what applications want it to be). An application sets desired: {"fan": "on"}. When the device connects, it receives the difference (the delta), turns the fan on and updates reported. Applications can always read the last known state, even when the device is asleep.

The rules engine

The rules engine is how data leaves IoT Core. A rule has an SQL-like query that runs on each incoming message and one or more actions that send the result somewhere:

SQL
SELECT topic(2) AS device, temperature, timestamp() AS ts
FROM 'sensors/+/data'
WHERE temperature > 30

Typical actions: write to DynamoDB, put the object in S3, send to a Kinesis stream or Firehose (see Kinesis), invoke Lambda (Lambda), publish to SNS, or republish to another MQTT topic. Rules run inside the managed service, so no servers are involved. An error action can capture messages that failed to deliver.

This stand-in runs a rule on a few messages, using the same idea: a topic filter, a condition and a projection.

Python A tiny rules-engine query

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The IoT family

  • AWS IoT Greengrass is a runtime that you install on an edge device or gateway. It runs Lambda-style components locally, filters and processes data near the machine, works while the network is down, and syncs when it returns. Use it when latency, bandwidth or connectivity make cloud-only processing impractical.
  • AWS IoT SiteWise collects data from industrial equipment, models assets (machines, production lines) and computes metrics such as overall equipment effectiveness. It suits factories.
  • AWS IoT Device Defender audits device configuration and detects abnormal behavior.
  • AWS IoT Device Management handles fleet registration, remote jobs and over-the-air updates.
  • AWS IoT Events and IoT Analytics have been used for event detection and analysis of device data. Some of these services have been retired or have announced end of support, so check their current status before choosing them. Many teams now store readings with Firehose and S3, query them with Athena, or keep time series in Amazon Timestream.

More products and what they are for

  • Onboarding and fleet management. IoT Core supports fleet provisioning, where a device with a claim certificate registers itself and receives its own identity. Fleet indexing then lets you search the fleet ("all devices with firmware 1.2 that reported in the last hour") and IoT Jobs push updates and commands to groups of devices.
  • AWS IoT Device Defender compares device behavior with a baseline. A bulb that suddenly sends ten times its normal traffic, or connects from an unexpected place, can raise an alert, which is how you detect compromised devices.
  • AWS IoT TwinMaker builds digital twins: it links data from IoT SiteWise, video and storage to a 3D model so that operators can see a building or plant, and the live values on each machine, in one scene.
  • AWS IoT FleetWise collects and shapes data from vehicles, and FreeRTOS and IoT ExpressLink give very small microcontrollers a secure path to AWS.
  • Device Advisor tests whether your firmware follows the IoT Core protocol and security rules before you ship it.

Typical stories: a connected smart lightbulb publishes its state to an MQTT topic and listens for commands through its shadow; a vineyard's sensors report soil moisture through a rule that triggers irrigation; a facility team views a whole building as a digital twin. The benefits are real-time insight and decisions, better customer experience (a vendor knows a device is failing before the customer calls) and lower cost from scalable, managed ingestion.

Python Pick an IoT service for a requirement

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A typical ingestion architecture

flowchart LR
    D["Devices<br/>X.509 + MQTT"] --> B["IoT Core<br/>broker + rules"]
    B -->|"all data"| F["Firehose"] --> S3[("S3 data lake")]
    B -->|"alerts"| L["Lambda"] --> N["SNS: notify team"]
    B -->|"latest value"| DDB[("DynamoDB")]
    S3 --> A["Athena / dashboards"]
    G["Greengrass gateway"] --> B

Raw data goes to the lake for analysis, alerts go to people, and the latest value goes to a fast table for an app.

Good habits

  • Give every device its own certificate and a narrow policy, never a shared one. A leaked certificate can then be revoked alone.
  • Use policy variables so one policy serves a whole fleet.
  • Keep payloads small and use compact topic names. You are billed by messages, and every message counts in 5 KB increments.
  • Plan for devices that reconnect all at once after an outage. Use backoff with jitter, as in high availability.
  • Monitor connection counts and rule errors in CloudWatch.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a device's X.509 certificate do in IoT Core?

    The certificate authenticates the device during the TLS handshake. Permissions come from the IoT policy attached to it.
  2. 2. Which topic filter matches sensors/greenhouse-3/temperature and sensors/greenhouse-4/temperature, but not sensors/greenhouse-3/humidity?

    The + wildcard matches exactly one level, so sensors/+/temperature matches any device but only the temperature topic.
  3. 3. What is a device shadow for?

    The shadow document holds reported and desired state. When the device reconnects it receives the delta and reports back.
  4. 4. Which component lets you filter, transform and route incoming messages to DynamoDB, S3, Lambda or Kinesis?

    Rules use an SQL-like query and actions to send messages to other AWS services.
  5. 5. When is AWS IoT Greengrass a good fit?

    Greengrass runs components on the edge device or gateway, which reduces latency and bandwidth and tolerates network outages.

Summary

  • IoT Core authenticates devices with X.509 certificates, authorizes them with IoT policies, and carries messages over MQTT (also HTTPS and WebSockets).
  • Topics are slash-separated paths. + matches one level and # matches the rest.
  • Device shadows keep reported and desired state so offline devices can still be managed.
  • The rules engine filters messages with SQL and sends them to S3, DynamoDB, Kinesis, Lambda, SNS and more.
  • Greengrass runs logic at the edge, SiteWise models industrial assets, and Device Defender and Device Management secure and operate a fleet.
  • Give each device its own certificate, keep payloads small and plan for reconnect storms.

Next: Project: Static Site on S3 and CloudFront