Skip to content
Tutorials Center
My learning

Topics

Lesson 57 of 64 ยท Projects

Project: Sensor and Time-Series Report

Turn gappy sensor readings into a report with pandas: resample to a regular grid, interpolate short gaps, flag anomalies with a rolling median and summarise by day.

Sensors misbehave. They skip readings, go offline for hours, and now and then report a number that is physically absurd. Yet someone still wants a daily report that says what happened and how much of it can be trusted.

In this project you build that report from a week of temperature readings taken every ten minutes in a greenhouse. The data is synthetic and uses a fixed seed, so the numbers on your screen are the same on every run. It contains the kinds of trouble real logs have: scattered missing readings, one long outage, and three glitches. You will practise the main time-series tools from this track: resampling, interpolation from Missing Data, rolling windows and daily aggregation.

The plan

flowchart LR
    A["raw readings<br/>(irregular timestamps)"] --> B["regularise<br/>resample to a 10-minute grid"]
    B --> C["fill short gaps<br/>interpolate up to 1 hour"]
    C --> D["detect anomalies<br/>rolling median residual"]
    D --> E["daily summary<br/>min, max, mean, quality"]
    E --> F["plot and report"]

Each stage hands a time-indexed Series or DataFrame to the next, and every stage can be checked on its own.

Step 1: look at the raw readings

The table has a time column and a temp_c column, one row per reading that actually arrived. The first question about any log is: is it regular? Subtract each timestamp from the previous one with diff() and count the distinct gaps. A perfect ten-minute sensor would show only 10min.

Python Step 1: how regular is the log?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Most steps are ten minutes, 25 are twenty minutes (a single missing reading), and one is a seven-hour hole. The temperature range is already suspicious: a greenhouse that swings between about 15 and 28 degrees over a day is believable, but we should look for single readings that do not fit.

Step 2: put the data on a regular grid and fill short gaps

Most time-series tools (rolling windows, resampling to other frequencies, comparing days) assume a regular grid. resample("10min").mean() creates one row per slot and leaves NaN where nothing arrived.

Then comes a decision that deserves to be written down. A single missing reading in a smooth signal is safe to interpolate: draw a straight line between its neighbours. Seven missing hours is not: a straight line across a night-to-morning change would invent data. So our rule is: fill gaps of up to six slots (one hour) and leave anything longer empty. interpolate(method="time") is aware of timestamps, and limit_area="inside" prevents guessing before the first or after the last reading. Because limit=6 would fill only the first six slots of a longer gap, which is misleading, the code measures each gap's length and blanks the long ones completely.

Python Step 2: regular grid and gap filling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The was_filled flag is important. It lets the report say how much of each day is measured and how much is estimated, so nobody mistakes an interpolation for a reading.

Step 3: find anomalies with a rolling median

What counts as "abnormal" when the temperature legitimately rises and falls by ten degrees every day? A global rule such as "more than three standard deviations from the mean" cannot work: the daily cycle itself is the main source of variation. Even a rolling mean and standard deviation misfire here, because the spread inside the window includes the cycle and any spike that is already in it.

A sturdier approach has three parts:

  1. Compute a baseline with a centred rolling median over one hour. A median ignores a single wild point, so the baseline follows the true curve.
  2. Compute the residual: reading minus baseline. On a healthy sensor it is tiny noise.
  3. Measure the typical size of residuals with a robust spread (the median absolute deviation, scaled by 1.4826 to be comparable to a standard deviation) and flag residuals larger than six times that.

The next example first shows why the simple rules fail, then applies the robust one.

Python Step 3: simple rules fail, a rolling median works

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The robust detector finds exactly the three glitches: two readings far above the curve and one far below. The two simple rules fail in opposite directions: the global rule sees nothing, the rolling z-score raises false alarms. Choosing the rule is the real work in anomaly detection, and you should always look at the flagged points before trusting a detector. A rolling window also looks at neighbours on both sides here (center=True), which is fine for a report written after the fact but would be lookahead in a live alarm. For real-time alerts, use only the past, as discussed in Rolling, Expanding and EWM.

Step 4: daily summary, quality columns and a chart

Now the report table. One row per day with the minimum, maximum and mean temperature, plus quality columns: how many real readings arrived, what percentage of the day that is, and how many anomalies were flagged. Before summarising, replace the flagged glitches with missing values; otherwise a single bad point becomes that day's maximum.

The chart has two panels. The top shows the cleaned series with the anomalies marked and the outage shaded. The bottom shows each day's range and mean.

Python Step 4: summary table and chart

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reading the table, six of the seven days are at least 96% complete, and one day (Wednesday) dropped to about 71% because of the outage. The daily means stay close to 21 degrees, but the Wednesday mean (20.35) is lower than the rest: the long gap removed hours of warm afternoon data, so that day's average is biased by the missing hours. A good report says so. Mean values of days with different completeness are not directly comparable.

Putting it together

The finished report is a pipeline of three functions: regularise, find_anomalies and the daily aggregation, exactly the structure from Building Data Pipelines. A few judgement calls are worth documenting in your own report:

  • Gap policy. Up to one hour was interpolated (25 slots); longer gaps stay empty. A different policy changes the means.
  • Detector settings. The one-hour window and the factor of six come from this sensor's noise. Another sensor needs its own look at the residuals.
  • Flagged is not wrong. The detector reports suspicious readings. A person (or a second sensor) decides whether they are faults or real events.
  • Resampling choice. mean was right for temperature. For rainfall or counts you would use sum, and for a status code last.

Ideas to extend the project

  • Add a second sensor with a different offset and compute the difference between the two; a sensor drifting away from its twin is a classic fault.
  • Use resample("h") to produce hourly means and compare them with the 10-minute series.
  • Add a rule that flags a "stuck" sensor: the same value repeated for more than 30 minutes (diff() == 0 with a rolling sum).
  • Test regularise with a hand-written five-row frame and assert_frame_equal, as in Testing and Validating DataFrames.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the project fill gaps of up to one hour but leave the seven-hour outage empty?

    Interpolation assumes the signal changes smoothly between neighbours. Over a long gap that assumption fails, so the honest choice is to leave the gap visible.
  2. 2. Why is a global z-score unable to find the glitches in this data?

    The legitimate daily cycle dominates the overall spread, hiding unusual points. Comparing each reading with a local baseline removes the cycle first.
  3. 3. What is the advantage of a rolling median over a rolling mean as the baseline?

    Means are pulled toward outliers; medians are not. That keeps the residual of a glitch large and the residuals of its neighbours small.
  4. 4. Why replace flagged anomalies with missing values before computing the daily maximum?

    Aggregations like max and min are very sensitive to single bad points. Masking them first keeps the summary meaningful, and the anomaly count is reported separately.

Summary

  • Check regularity first with diff(); then resample("10min").mean() puts readings on a regular grid.
  • Interpolate short gaps with interpolate(method="time"), leave long ones empty, and keep a was_filled flag.
  • Detect anomalies against a local baseline (rolling median) using a robust spread, not a global z-score.
  • Centered windows are fine for after-the-fact reports but are lookahead in live monitoring.
  • Mask anomalies before taking daily min, max or mean, and report completeness next to every daily number.
  • Document gap policy and detector settings, since they change the results.

Next: Project: Customer Segments with RFM