Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 37 ยท Projects and Practice

Project: Sensor Time Series

Clean noisy sensor readings with Polars. Fill missing hours, interpolate gaps, resample to daily summaries, flag anomalies with rolling statistics and compare sensors.

Time series data (readings that arrive with timestamps) is where DataFrame skills pay off most: temperature sensors, server metrics, stock prices, website visits. They share the same problems. Readings are missing, there are spikes that are errors rather than events, and what you want to know is on a coarser scale than the data (daily averages, weekly trends).

In this project you will take 30 days of hourly temperature readings from three sensors and walk the full cleaning and analysis path: find and fill missing hours, interpolate short gaps, resample to daily summaries, detect anomalies with rolling statistics, and compare sensors with correlation. All data is synthetic and generated with a fixed seed, so you get the same results each time.

Step 1: generate the readings

Each sensor follows a daily temperature cycle (warmer in the afternoon) plus noise. We then delete about 4% of the readings to simulate dropouts, and inject a few false spikes.

Python Generate three sensors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three sensors times 720 hours would be 2,160 rows; the missing ones are the dropouts. Notice the data is in long format: one row per reading, with a sensor column that identifies the series. That is the shape Polars' time functions expect.

Step 2: find the gaps

Before fixing anything, measure it. The gap between consecutive readings inside each sensor tells you where hours are missing.

Python Detect missing hours

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

diff().over("sensor") computes the time since the previous reading separately for each sensor. Anything over one hour is a gap, and its size tells you how many readings are missing.

Step 3: put every hour back, then interpolate

DataFrame.upsample inserts the missing timestamps (with null values) at a regular frequency. With group_by each sensor gets its own complete range. Afterwards interpolate() fills short gaps linearly between the neighbours.

Python upsample and interpolate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two details. First, upsample fills sensor with null for the new rows, so we forward-fill that identifier. Second, linear interpolation is reasonable for slowly-changing quantities like temperature over a gap of an hour or two. For longer gaps, interpolating invents data; keep the original temp column next to temp_filled so you can always tell measured values from estimated ones, and consider leaving long gaps as null.

Step 4: resample to daily summaries

The sensors report hourly, but reports want days. group_by_dynamic with every="1d" and group_by="sensor" gives per-sensor daily statistics. The frame must be sorted by the time column within each group.

Python Daily min, max and mean per sensor

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the high and low for the roof: a day with an absurd value is an injected spike, and the daily mean is pulled by it. That is the cue for the next step.

Step 5: flag anomalies with rolling statistics

A good definition of "anomalous": much further from the recent typical value than normal readings are. For each sensor we compute a rolling median over 7 readings centered on the current one (a median is robust, because one spike hardly moves it), take the residual (reading minus that median), and flag residuals that are more than three times larger than what the sensor's residuals normally reach (their 95th percentile).

Python Detect spikes with a rolling median

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The recipe: smooth (rolling median), measure the residual, scale it with a robust statistic, and threshold. Setting flagged values to null and interpolating gives a cleaned series, and the maximum drops from the false 44 degrees to a believable value. Always look at what you flagged: a handful of rows are real spikes, but a flag on the very last reading of a sensor is an edge effect, because the centered window there has fewer neighbours and is less reliable. The multiplier (3) is a judgement call: lower finds more, but also raises false alarms. We used the 95th percentile rather than a median-based scale because for smooth signals the median residual is often exactly zero, which would make every reading look anomalous.

Step 6: compare the sensors

Does the garden follow the roof? Pivot to a wide table with one column per sensor, then use pl.corr for the correlation. We average to daily values first, so the comparison is not dominated by the daily cycle.

Python Wide table and correlations

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The roof and the garden share the same daily rhythm, so they correlate strongly, while the cellar hardly moves and correlates weakly with the sun-driven sensors. The "average day" table (mean temperature per hour) confirms it with numbers: the roof swings by many degrees, the cellar by less than one.

Extensions to try

  • Use rolling_mean_by("ts", window_size="6h") to compute a time-based moving average (it handles missing hours correctly, unlike a row-count window).
  • Compute day-over-day change of each sensor's daily mean with diff().
  • Find the longest run of hours above 25 degrees with the "flag, cumulative sum of changes" recipe from the log analysis project.
  • Join a small table of sensor locations and report by location.
  • Write the cleaned data to Arrow IPC and read just one sensor back (IO chapter).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does DataFrame.upsample(time_column='ts', every='1h') do?

    upsample makes the time axis regular by adding the missing timestamps; other columns are null there until you fill them.
  2. 2. Why keep the original column next to an interpolated one?

    Interpolated numbers are estimates. Keeping the raw column preserves what was actually measured.
  3. 3. Why use a rolling median rather than a rolling mean for spike detection?

    The median is robust to outliers; a rolling mean would be dragged towards the spike and hide it.
  4. 4. What is required before group_by_dynamic with group_by='sensor'?

    Dynamic windows scan the time column in order, so sort by the group key and time first.

Summary

  • Keep time series in long format with a series key (sensor) and a sorted timestamp.
  • Measure gaps with diff().over(key); fix them with upsample, forward_fill and interpolate (per group).
  • Resample with group_by_dynamic(every=..., group_by=...).
  • Detect anomalies with a robust recipe: rolling median, residual, robust scale, threshold; then null and interpolate.
  • Compare series by pivoting to wide format and using pl.corr.

Next: Practice: Exercises with Solutions