Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 36 ยท Edge, Cloud and Data Pipelines

IoT Data Pipelines: From Sensor to Insight

Follow IoT data from sensor to dashboard: payload formats, ingestion, validation, deduplication, normalization, enrichment and storage, with a runnable pipeline.

A sensor produces a number. A business wants an answer: is the cold room safe, is the pump about to fail, how much energy did the building use? Between the two sits a data pipeline, a chain of stages that receives raw readings, cleans them, stores them and turns them into something useful. Real IoT data is messy: readings arrive late, twice, out of order, in different units, or not at all. A good pipeline expects that.

This lesson walks through the stages and builds a small pipeline you can run. The stages feed edge processing and time-series storage in the next lessons.

The stages

A typical pipeline

  1. Device
Measures and encodes a payload
  1. Ingest
Broker or gateway receives it
  1. Clean
Validate, dedupe, convert units
  1. Store
Time-series database or lake

Then analyze, alert and visualize

1 / 2
flowchart LR
  D["Devices"] --> G["Gateway / Broker"]
  G --> V["Validate & normalize"]
  V --> S[("Storage")]
  V --> R["Rules and alerts"]
  S --> A["Analytics & dashboards"]

Many pipelines split after cleaning: a hot path for alerts and a cold path for history.

Payload formats

What the device sends is its payload. Common choices:

  • Plain text: 21.5. Tiny, but no unit, time or device ID.
  • JSON: {"id":"s1","t":21.5,"ts":1700000000}. Readable and flexible, but verbose.
  • CSV-like: s1,21.5,1700000000. Compact, fragile when fields change.
  • Binary (CBOR, Protobuf, custom): smallest; used on LoRaWAN or cellular where every byte costs money.

Whatever you choose, include a timestamp, a device ID and a schema version. Devices without a real-time clock should send their uptime counter and let the gateway stamp the arrival time.

Validate, deduplicate, normalize

The example below pipes eight raw messages through four small functions. Study what each stage rejects and why.

Node.js A four-stage cleaning pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Four of the eight messages were rejected, and one was converted. The late reading (ts 90) was kept and sorted into place: event time (when it was measured) matters more than arrival time (when it reached you). Keep rejected messages in a dead-letter store instead of discarding them silently, so you can debug a faulty sensor or a firmware bug later.

Enrichment and aggregation

Raw readings rarely say what a device is or where it sits. Enrichment joins each reading with metadata from a device registry: location, owner, calibration offset, warning limits. Then aggregation reduces volume: one reading per second becomes one average per minute.

Node.js Enrich and aggregate into one-minute buckets

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Keeping min and max along with avg is cheap and preserves the spikes that an average would hide.

Design questions to ask

  1. How fresh must the answer be? Alerts need seconds (stream processing); monthly reports can wait (batch).
  2. What if a message arrives twice? Make every stage idempotent or deduplicate by device + timestamp.
  3. What if the pipeline is down? Use a durable queue (a broker with persistence, Kafka, a cloud queue) between stages and let devices buffer locally.
  4. How long to keep raw data? Raw per-second data is expensive. Keep it for days, keep aggregates for years.
  5. Who can read what? Apply access control before data fans out.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use the measurement timestamp (event time) instead of the arrival time?

    Network delays and offline buffering mean arrival order differs from measurement order.
  2. 2. What is the best treatment for a reading of -999 from a temperature sensor?

    Such codes mean a sensor error; they must not pollute aggregates, but they are evidence of a fault.
  3. 3. Why keep min and max in an aggregate besides the average?

    A single spike barely moves a one-minute average but may be exactly what you need to see.
  4. 4. Which step removes the effect of QoS 1 retransmissions?

    Duplicates share the same device ID and timestamp, so they can be recognized and dropped.

Summary

  • A pipeline moves data device, ingest, clean, store, analyze; messy input is the normal case.
  • Send device ID, timestamp and schema version; choose a payload format by cost (JSON for flexibility, binary for tight links).
  • Validate, deduplicate and normalize early; keep rejects in a dead-letter store.
  • Enrich with registry metadata and aggregate to control volume, keeping min/max.
  • Decide freshness, durability and retention before building.

Next: Edge Computing: Filtering and Deciding Near the Device