Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 64 ยท Time Series

Rolling, Expanding and EWM

Learn pandas rolling(), expanding() and ewm() to smooth noisy data, build moving averages, use time-based windows and avoid look-ahead mistakes.

A single day's number is often noise. One cold morning does not make a cold month, and one huge order does not make a trend. To see the signal, you look at a window: the average of the last seven days, the highest price this quarter, the total since the year began. pandas has three families of moving windows, and all of them work as "slide a window along the data and compute something for each position".

  • Rolling windows have a fixed size (3 rows, or 7 days) and slide forward one step at a time.
  • Expanding windows start at the first row and grow, so every result uses all history up to that row.
  • Exponentially weighted (EWM) windows use all history too, but recent values weigh more than old ones.

Unlike resample() (see Resampling and Time Windows), which collapses many rows into one, these windows produce one result per input row, so the output lines up with your original index and you can add it as a new column.

Rolling windows

rolling(window) returns a window object; you then call an aggregation such as mean(), sum(), std(), min(), max(), median() or count().

flowchart LR
    subgraph data["temp (rows 1 to 6)"]
        r1["14.0"] --- r2["15.5"] --- r3["13.0"] --- r4["18.0"] --- r5["21.5"] --- r6["19.0"]
    end
    r3 -. "window 1: rows 1-3" .-> o3["mean = 14.17"]
    r4 -. "window 2: rows 2-4" .-> o4["mean = 15.50"]
    r5 -. "window 3: rows 3-5" .-> o5["mean = 17.50"]

A window of size 3 slides one row at a time. The result is attached to the last row of each window, so the first two rows have no complete window.

Python A 3-day moving average

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the table carefully, because every column teaches something:

  • mean3 is NaN for the first two rows. A window needs window observations by default; until it has them, the result is missing.
  • min_periods=1 relaxes that rule. The first value is just the first temperature, the second is the average of two. Use it when a shorter window at the start is acceptable, but remember those early values are less reliable.
  • center=True places the result in the middle of the window instead of at its end. That gives a nicely aligned smoothed curve for plotting, but each value then uses data from after its own date (see the look-ahead warning below).
  • std3 is the sample standard deviation (with ddof=1), so a window of three needs at least two distinct values to be non-zero.

Time-based windows

If your rows are not evenly spaced, "the last 3 rows" and "the last hour" mean different things. Give rolling() an offset string such as "1h" or "7D" and the window becomes a time span. This needs a DatetimeIndex (or on="column"), sorted ascending.

Python Row count versus time span

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With "1h", the window for 10:30 holds only the reading at 10:30 itself (09:25 is more than an hour earlier), so the "average" is 30. The count() output shows how many readings backed each number. A time-based window includes its right edge and excludes its left edge: the window for 10:45 covers (09:45, 10:45]. Time windows have no min_periods minimum by default, so check count() before trusting a value computed from a single reading.

Expanding windows

expanding() is a rolling window whose start is pinned to the first row. It answers "what is true so far?": running totals, the all-time high, the average to date.

Python Expanding and EWM

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

expanding().sum() matches cumsum(), and expanding().max() matches cummax(); the cumulative methods are faster, but expanding() also supports mean(), std(), median() and custom functions, which cumsum family does not.

Exponentially weighted windows

A plain rolling mean treats the oldest value in the window exactly like the newest, and then forgets it entirely when it slides out. An exponentially weighted moving average (EWMA) never forgets abruptly: each new value gets weight alpha, and everything before it is multiplied by 1 - alpha. A big alpha reacts fast; a small alpha is smooth and slow.

You can choose the strength in several equivalent ways:

Argument Meaning
alpha=0.5 smoothing factor between 0 and 1
span=3 like a window of 3 periods (alpha = 2 / (span + 1))
halflife=2 the number of periods after which a value's weight halves
com=1 "center of mass" (alpha = 1 / (1 + com))

In the example above, ewm(alpha=0.5) reaches 124.52 after the big 150 month, while the 3-month rolling mean says 120.00. The smaller alpha=0.2 gives a calmer 113.26. Exponential weighting is popular for forecasting baselines and monitoring, because it needs no window size and no stored history beyond the previous result.

Custom functions with apply

When no built-in aggregation fits, pass your own function to rolling(...).apply(func). It receives each window and must return a single number. Use raw=True so the function gets a plain NumPy array instead of a Series: it is much faster and is almost always what you want.

Python Custom window functions and look-ahead

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Custom apply is slow compared with built-ins, because it calls Python once per window. Prefer mean, std, quantile, min, max and friends when they do the job.

Pitfall: looking into the future

A rolling feature is only legitimate if it uses information that was available at that moment. Two habits break that:

  1. center=True uses values after the current row. It is fine for drawing a smooth line on a chart, but never for a feature that you would compute in real time (for example when predicting tomorrow's sales).
  2. Including the current row in its own baseline. A spike pulls the window mean and standard deviation toward itself, which hides how unusual it was. In the example above the leaky z-score is a tame 2.2, while the honest one, computed from the seven days before the spike, is far above any sane threshold. Use .shift(1) after the rolling aggregation, or closed="left" on a time-based window, to exclude the current row.

Windows follow row order, not dates. For rolling(3), "the previous 3 rows" can mean three days or three months depending on how you sorted the data or where gaps are. Sort the index first, and if rows are irregular use a time offset like "7D". Missing values inside a window also make the result NaN unless you lower min_periods.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does s.rolling(3).mean() return for the first two rows of the series?

    By default min_periods equals the window size, so incomplete windows give NaN. The result keeps the same length as the input.
  2. 2. Which call gives you the highest value seen so far at every row?

    expanding() pins the window start to the first row, so each row's maximum covers all history up to that row. s.max() returns one number.
  3. 3. Why does rolling('1h') need a sorted DatetimeIndex (or on=...)?

    A time-based window contains every row within the last hour, which is meaningful only when timestamps are known and monotonic. Rows do not have to be evenly spaced.
  4. 4. A bigger alpha in ewm(alpha=...) means...

    alpha is the weight of the newest value. A large alpha makes the newest values dominate (fast reaction); a small alpha makes the curve smooth and slow.

Summary

  • rolling(n) slides a fixed window one row at a time; the result sits on the last row of each window and starts with NaN until the window is full (change this with min_periods).
  • rolling("7D") uses a time span, which is the right choice when rows are unevenly spaced.
  • expanding() grows from the first row (running totals, all-time highs); ewm() weights recent values more through alpha, span, halflife or com.
  • center=True is for plotting only, because it reads future values.
  • Use rolling(...).apply(func, raw=True) for custom logic, but prefer built-in aggregations for speed.
  • To avoid look-ahead, keep the current row out of its own baseline with .shift(1).

Next: Time Zones explains how to keep timestamps correct when your data crosses regions and daylight saving changes.