Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 64 ยท Time Series

Time Series Basics

Learn the basics of time series in pandas. Use DatetimeIndex, select by date strings and slices, shift and diff, rolling date windows, PeriodIndex and frequency strings.

A time series is a set of values recorded over time: daily visitors, hourly temperatures, monthly revenue. pandas was born in finance, where almost every table has a date, and it still has the best time handling of any mainstream data library. The key idea is simple. When the index of a Series or DataFrame is made of timestamps, pandas understands that the rows have an order and a spacing, and a lot of operations become short and readable: select a month with "2024-02", move values one day later with shift, or average over a seven-day window.

This lesson is about the foundations: the DatetimeIndex, selecting by date, shifting and differences, time-based windows, Period objects and frequency strings. If to_datetime and the .dt accessor are new to you, read Dates and Times first. The following lessons build on this one with resampling, rolling windows and time zones.

A DatetimeIndex

pd.date_range() creates a regular sequence of timestamps. Give it a start, plus either an end or a number of periods, and a frequency freq ("D" is calendar days, which is the default). Use it as the index of a Series, and you have a time series. The example uses fixed-seed random numbers so every run gives the same data.

flowchart LR
    A["date_range<br/>start, periods, freq"] --> B["DatetimeIndex"]
    C["values"] --> D["Series / DataFrame"]
    B --> D
    D --> E["select, shift, window,<br/>resample"]

A DatetimeIndex plus values gives a time series that supports date-aware selection and operations.

Python Build a daily series and select by date

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three things deserve attention here.

Date strings are flexible. "2024" selects a whole year, "2024-02" a month, "2024-02-10" one day. This is called partial string indexing, and it works on the index with [] for a Series and with .loc for a DataFrame.

Slices include both ends. Positional slices like s[0:3] exclude the end, but label slices, including date labels, include it. "2024-01-10":"2024-01-13" gives four days. A slice end of "2024-01-13" includes the whole day, even when the index has timestamps with times of day.

The index must be sorted for slicing. Slicing needs a monotonic (sorted) index. If your data arrives in random order, call sort_index() first. On an unsorted index, a slice between dates that are not present as exact labels raises an error.

Using date parts to filter

The index also exposes date parts: index.year, index.month, index.day, index.dayofweek (Monday is 0) and index.day_name(). Combine them with boolean masks, for example weekends: visits[visits.index.dayofweek >= 5]. For a column of dates you would use the .dt accessor instead; once the dates are the index, you read them straight from it.

If your dates are in a column, move them into the index with set_index, and keep the data sorted:

Python
df["when"] = pd.to_datetime(df["when"])
df = df.set_index("when").sort_index()

Shifting, differences and changes

Many time-series questions compare a value with an earlier one. Three methods cover most of them:

  • shift(n) moves the values n rows down (later in time); the first n entries become NaN. Negative n looks into the future.
  • diff(n) is the value minus the value n rows earlier.
  • pct_change(n) is the relative change: (x - x.shift(n)) / x.shift(n).

Because they work by row position, they assume regular spacing. With gaps in the data, "one row earlier" may mean three days. When you want to move by time rather than by rows, pass freq: visits.shift(freq="7D") keeps all values and moves the index forward by seven days.

Windows that follow the calendar

A rolling window can be defined by a number of rows (rolling(7)) or by a time span (rolling("7D")). The time-based version needs a DatetimeIndex and handles missing days properly: it always covers the last seven days, however many rows they contain. It also starts producing values immediately (the first window has just one row), while the row-based version shows NaN until seven rows exist. The next lessons dive deeper; here you get a first look at the idea.

Python Shift, diff and rolling windows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the first row of yesterday, change and pct is NaN: there is no earlier value. Notice also that the 7-row rolling mean is empty for the first six rows, while the 7-day mean averages whatever exists so far (92.5 on January 2 is the mean of the first two days). When the series has no gaps the two agree from the seventh day on. If days were missing, the row-based version would silently cover a longer period.

Periods: spans of time

A Timestamp is a point in time. A Period is a span: "March 2024", "the week starting Monday", "Q3 2025". Periods are useful for reporting, because every date belongs to exactly one period, and period arithmetic is exact (March + 1 is April, whatever the number of days).

  • index.to_period("M") converts timestamps to monthly periods.
  • pd.period_range("2024-01", periods=3, freq="M") builds a PeriodIndex.
  • period_index.to_timestamp() converts back, using the start of each period.
  • Period.asfreq("M") converts a period to a coarser or finer frequency.

A very common idiom is grouping by period:

Python Periods and monthly totals

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For plain monthly totals you will usually use resample("ME").sum(), which keeps timestamps (see Resampling and Time Windows). The Period route is handy when you want labels such as 2024-01 and exact period arithmetic.

Frequency strings

The freq argument takes short codes. A number in front multiplies them ("2D", "15min"). The most useful:

Code Meaning
"D" Calendar day
"B" Business day (Monday to Friday)
"h", "min", "s" Hour, minute, second
"W" or "W-MON" Weekly, anchored on Sunday or on a named weekday
"MS" / "ME" Month start / month end
"QS" / "QE" Quarter start / quarter end
"YS" / "YE" Year start / year end

pandas renamed several aliases: month end is "ME" (not "M"), quarter end "QE", year end "YE", and hours and minutes are lowercase "h" and "min". The old forms (M, H, T, Y, Q) raise an error in pandas 3, so use the new ones.

Python Frequencies with date_range

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Sorting and checking a time series

Before analysing any time series, check three things:

  1. Sorted? s.index.is_monotonic_increasing. If not, s = s.sort_index().
  2. Unique? s.index.is_unique. Duplicated timestamps may be legitimate (two sensors) or a bug.
  3. Regular? s.index.to_series().diff().value_counts() shows the gaps between rows. A regular daily series has only 1 days.

s.index.freq is None when pandas cannot infer a regular frequency, for instance after filtering out rows. Some operations, such as asfreq and resample, can restore it by inserting the missing timestamps.

Do not mix up shift (moves values, keeps the index) with shift(freq=...) (moves the index, keeps the values). Also remember that shift, diff and pct_change count rows. On a series with missing days, they compare the wrong dates.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does visits["2024-02"] return for a Series with a daily DatetimeIndex?

    Partial string indexing selects every row that falls inside the given month. "2024" would select the whole year and "2024-02-10" a single day.
  2. 2. How many rows does visits["2024-01-10":"2024-01-13"] return on a daily series?

    Label-based slices, including date slices, include both endpoints. The result has the 10th, 11th, 12th and 13th.
  3. 3. What is the difference between rolling(7) and rolling("7D")?

    The integer window counts rows, so gaps in the data stretch the period. The time-based window always covers seven days of the index.
  4. 4. Which code gives the monthly Period for each timestamp in an index?

    to_period("M") converts timestamps into monthly Period objects such as 2024-01. index.month only returns the month number, which loses the year.
  5. 5. Which frequency string means month END in current pandas?

    "ME" is month end and "MS" month start. The old alias "M" no longer works in pandas 3.

Summary

  • A DatetimeIndex turns a Series or DataFrame into a time series; build one with pd.date_range or set_index on a datetime column, then sort_index().
  • Select with date strings: "2024", "2024-02", "2024-02-10", and slices that include both ends.
  • shift, diff and pct_change compare rows by position; shift(freq=...) moves the index; time-based windows like rolling("7D") follow the calendar.
  • A Period represents a span such as a month; use to_period for reporting and to_timestamp to go back.
  • Frequency strings (D, B, h, W-MON, MS, ME, QS, YS) describe spacing; check sortedness, uniqueness and regularity of any time series.

Next: Resampling and Time Windows