Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 40 ยท Getting Started

Missing Data and Resampling

Fix gaps and irregular timestamps in time series: find missing dates, fill with ffill, interpolation or seasonal values, and resample to coarser or finer frequencies.

Real time series are messy. Sensors drop out, shops close on holidays, logs arrive at random moments. Most forecasting methods need a clean, regular grid with no holes, so the first practical job is turning raw records into that grid. Two tools do almost all of it: resampling (changing the time step) and filling (deciding what to put in the gaps).

Find the gaps first

A regular series should have exactly one row per step. Compare the index with the full range you expect.

Python Detect missing timestamps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

reindex against a complete range turns missing rows into missing values, which is the form every filling method expects. Also check for duplicated timestamps with s.index.duplicated().sum(); those must be aggregated or removed before reindexing.

Filling strategies

There is no universal best fill. The right one depends on what the series means.

Method How Good for Danger
Forward fill repeat the last known value prices, settings, status long gaps become flat lines
Backward fill use the next known value rare uses the future
Linear interpolation straight line between neighbours smooth physical signals invents a smooth path
Time interpolation linear in elapsed time, for uneven steps irregular data same
Seasonal fill value from one cycle earlier strongly seasonal data copies noise
Zero fill with 0 counts where no record means none wrong if it means unknown
Python Compare filling methods

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at 3 and 4 May: forward fill repeats 12, backward fill jumps to 15, and linear interpolation walks 12, 13, 14, 15. For a temperature sensor linear is sensible; for a stock price ffill is the honest choice, because the last quote is still the latest known price.

Never use backward fill or centered interpolation on data you will later forecast with. Both use values from after the gap. That is legitimate for cleaning a historical record, but if the filled values feed a model evaluated on the same period, you leak the future.

Seasonal filling

For data with a strong repeating pattern, copying the value from one cycle earlier beats a straight line. Here we have a weekly pattern with a missing Wednesday.

Python Fill from the same weekday last week

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Linear interpolation averages the two neighbouring days (120 and 110), giving a bland 115, while the seasonal fill recovers the Wednesday peak of 130 exactly in this noise-free example. Real data has noise, so the seasonal value would be close rather than exact, and an average of several previous cycles is safer.

Resampling: changing the step

resample is groupby for time: it buckets rows into regular periods and you choose how to aggregate.

  • Downsampling goes from fine to coarse (hourly to daily). You must aggregate: sum for flows such as sales, mean for levels such as temperature, last for stocks such as account balance.
  • Upsampling goes from coarse to fine (monthly to daily). It creates new rows that you fill.
Python Downsample hourly data to daily

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

kWh consumed adds up, so sum is the meaningful daily total; mean shows the average level. Always check count to see whether a bucket is complete: a day with only three readings has a tiny sum for the wrong reason.

Python Upsample monthly to daily

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A monthly budget that applies to every day of the month is a step function, so ffill is right. A smoothly changing measurement would use interpolate. The output ends at 1 March because there is no data after the last point to define later days.

Irregular events to a regular series

Event data, such as one row per purchase, becomes a time series by counting or summing per period. Empty periods become zeros here because "no purchase" really means zero.

Python Events to a daily count

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

3 and 4 August (a weekend) have no events and show 0. Choosing zero here is a modelling decision you should be able to defend.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How do you turn missing rows (absent dates) into explicit NaN values?

    reindex with a full range adds the absent dates as rows with NaN, ready to be filled.
  2. 2. Hourly rainfall in millimetres is resampled to daily. Which aggregation is right?

    Rainfall is a flow that accumulates, so the daily total is the sum of the hourly amounts.
  3. 3. Why can backward fill be dangerous before forecasting?

    Backward fill takes the next known value, so earlier rows contain information that was not yet available.
  4. 4. A monthly price list applies for the whole month. Which upsampling to daily is most appropriate?

    The price is constant within the month, so repeating the last known value matches reality.

Summary

  • Compare the index with a complete date_range to find gaps and use reindex to expose them.
  • Choose fills by meaning: ffill for states, interpolation for smooth signals, seasonal fill for patterned data, zero for true "none".
  • Avoid backward fill and centered methods when the data will be used to forecast.
  • Downsampling needs an aggregation chosen by the nature of the variable; upsampling needs a fill rule.
  • Check bucket counts after resampling.

Next: The Forecasting Workflow and Baselines