Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 40 ยท Getting Started

The Datetime Index in pandas

Master the pandas DatetimeIndex for time series: parse dates, build date ranges, slice by date strings, extract calendar parts, shift, diff and compute rolling statistics.

pandas was built with time series in mind, and almost everything in this course rests on one object: the DatetimeIndex. Once your dates are the index, you can slice with date strings, group by month, shift values by a period and resample to another frequency with one line each.

This lesson covers the toolkit you will use in every later chapter: parsing, ranges, selection, calendar fields, shifting and differencing, and rolling windows.

Parsing dates

Data usually arrives as text. pd.to_datetime converts it, and set_index makes it the index. Give an explicit format when you can: it is faster and avoids day/month confusion (03/04/2024 is March 4th in the US and April 3rd in most of the world).

Python From text to DatetimeIndex

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how the same text produces two different dates. When the format is wrong or the text is dirty, use errors="coerce" to turn bad values into NaT (the datetime version of NaN) instead of raising an error.

Building regular ranges

pd.date_range creates a regular index from a start, an end or a number of periods, plus a frequency code.

Code Meaning Code Meaning
D calendar day h hour
B business day W week (Sunday)
MS month start ME month end
QS quarter start YS year start
Python Date ranges and frequency codes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Month start (MS) and month end (ME) are different anchors, and mixing them is a classic source of off-by-one-month bugs when you join two series.

Selecting by date

With a sorted DatetimeIndex, .loc accepts partial date strings: a year, a month, or a range.

Python Slicing with date strings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Unlike ordinary Python slicing, label slices on a date index include both ends.

Calendar fields

The .dt accessor (on a column) or the index attributes (on an index) expose parts of the date. These become features in the machine learning chapter and handy group keys now.

Python Group by calendar parts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The weekend effect is obvious in the grouped means; this "day of week" pattern is exactly what seasonality looks like in daily data.

Shifting and differencing

Two operations appear constantly:

  • shift(k) moves values down by k rows, so each row sees the value from k steps earlier (a lag).
  • diff(k) is the value minus its k-step lag, the change over k periods. pct_change(k) gives the relative change.
Python Lags, differences and growth

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first row has NaN in the derived columns because there is nothing before it. Be careful: shift(-1) brings the future into the current row. That is fine for building a target column, but it is a leak if it ever ends up in a feature.

Rolling windows

A rolling window computes a statistic over the last k observations, row by row. It is the basis of moving averages (later in this chapter).

Python A 3-month rolling mean

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

By default a window needs all k points, so the first k-1 results are NaN. min_periods relaxes that.

Use a trailing window (the default) when forecasting. A centered window (center=True) peeks at future values and is only for description.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does errors="coerce" do in pd.to_datetime?

    Unparseable entries become NaT (not a time) so that you can find and handle them instead of getting an exception.
  2. 2. Which frequency code gives the first day of each month?

    MS is month start; ME is month end. Mixing them shifts dates by a month when you join series.
  3. 3. What does s.shift(2) do?

    shift(k) lags the series by k rows and leaves NaN at the start.
  4. 4. Why is a centered rolling window risky for forecasting features?

    A centered window includes points after the current one, which would not be known at prediction time.

Summary

  • Parse dates explicitly with pd.to_datetime(..., format=...) and make them the index.
  • pd.date_range builds regular indexes; know D, B, h, MS, ME, QS, YS.
  • Partial date strings slice a sorted index, with both ends included.
  • .dt and index attributes give calendar parts; shift makes lags, diff and pct_change make changes.
  • Rolling windows are trailing by default; keep them that way when predicting.

Next: Missing Data and Resampling