Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 64 ยท Selecting and Filtering

The Index

Understand the pandas Index, use set_index and reset_index, name indexes, reindex and align data by label, deal with duplicate labels, and choose index or column.

Every Series and DataFrame has an index: the labels attached to its rows. It looks like a column of row numbers, but it does more. The index is what loc searches, what arithmetic between two objects lines up on, and what survives when you filter or sort. Treating it as a first-class part of your data, rather than as decoration, makes many tasks shorter and safer.

This lesson explains the default RangeIndex, how to promote a column to the index and back, how to name and replace the index, how reindex and label alignment work, what duplicate labels do, and when an index is better than a plain column. It builds on loc and iloc.

The default index

If you do not provide an index, pandas creates a RangeIndex: the integers 0, 1, 2, .... It stores only a start, a stop and a step, so it uses almost no memory, and it is what you get from read_csv, from most constructors and from reset_index.

Python Inspecting an index

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

An index is an immutable object. You cannot change one label in place (df.index[0] = "x" raises a TypeError); you replace the whole index instead.

set_index and reset_index

set_index("col") moves a column into the index. reset_index() does the reverse: the index becomes an ordinary column and a fresh RangeIndex takes its place. Pass drop=True to throw the old index away instead of keeping it as a column.

flowchart LR
    A["Columns: id, name, score<br/>index: 0, 1, 2"] -->|"set_index('id')"| B["Columns: name, score<br/>index: id"]
    B -->|"reset_index()"| A
    B -->|"reset_index(drop=True)"| C["Columns: name, score<br/>index: 0, 1, 2"]

set_index and reset_index move labels between the index and the columns.

Python set_index and reset_index

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A filtered or sorted DataFrame keeps its old labels, which is useful for tracing rows back but untidy for display. reset_index(drop=True) renumbers them. Setting an index does not require uniqueness, but loc on a duplicate label returns several rows (see below).

Names and renaming

An index can have a name, which appears in the printout and becomes the column name after reset_index. Set it with rename_axis or by assigning index.name. Change the labels themselves with rename, or replace the whole index by assigning to df.index.

Python Naming and replacing labels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Replacing labels with set_axis

When you have a complete list of new labels, df.set_axis(labels, axis=0) returns a copy with that list as the row index, and axis=1 (or "columns") replaces the column names. The list must have exactly as many labels as the axis has entries. It returns a new object, so it fits in a method chain, unlike assigning to df.index.

Python set_axis and resetting levels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a multi-level index, reset_index() moves all levels back into columns; level= picks some, and drop=True discards instead of keeping. This is the usual last step after a groupby on several keys, because it gives a flat table again.

reindex: choose the labels you want

reindex(new_labels) builds a new object with exactly those labels, in that order. Labels that exist keep their data; labels that did not exist get NaN (or a fill_value). Labels not in the new list are dropped. This is the standard way to force a complete, ordered set of labels, such as every weekday or every product code.

Python reindex

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Unlike loc, reindex never raises for a missing label; it fills. That makes it the safe choice when a label might be absent.

Alignment: the index in arithmetic

When you combine two Series or DataFrames, pandas lines the values up by label, not by position. Labels present in only one object give NaN. This is powerful and occasionally surprising.

Python Label alignment

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If you ever add two columns and get unexpected NaN values or numbers that seem shifted, compare the indexes first. Alignment also explains why df["new"] = other_series fills NaN when the indexes differ.

Duplicate labels

An index does not have to be unique. That is sometimes meaningful (several orders per customer), but it changes what loc returns: one match gives a row, several matches give a DataFrame. Some operations, such as reindex, refuse to work on duplicates.

Python Duplicate index labels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When a column is supposed to be a unique key, check result.index.is_unique right after set_index (or df["col"].is_unique before it), and use duplicated() to find the offenders. The old verify_integrity argument of set_index is deprecated in pandas 3.

Index or column?

Both are fine places for an identifier. A rough guide:

  • Use the index for a unique key you look up often (df.loc[customer_id]), for dates in a time series (see Time Series Basics), and for labels you want arithmetic to align on.
  • Keep it a column when the values are ordinary data you filter, group or aggregate, when they repeat, or when you will write the table to CSV and want every field visible. Many functions (groupby(..., as_index=False), merge(on=...), plotting) work naturally with columns.
  • Converting back and forth is cheap, so do not agonize: use set_index for lookups or alignment, then reset_index before output.

A hierarchical index with several levels, such as country then city, is covered in MultiIndex. Everything in this lesson (labels, alignment, reindex) carries over.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does df.reset_index(drop=True) do?

    reset_index() alone turns the old index into a column. With drop=True the old labels are simply discarded.
  2. 2. a = Series([1, 2], index=['x', 'y']) and b = Series([10, 20], index=['y', 'z']). What is (a + b).tolist()?

    Addition aligns by label. Only y appears in both, giving 12; x and z have no partner and become NaN.
  3. 3. What happens when you reindex a Series with a label that does not exist in it?

    reindex conforms the object to the labels you give, filling missing ones. That is the difference from loc, which raises KeyError.
  4. 4. Which is true about an index with duplicate labels?

    Duplicates are allowed. A label with several matches returns all of them.

Summary

  • The index holds the row labels; by default it is a RangeIndex of integers, and it is immutable (replace it, do not edit it).
  • set_index("col") promotes a column; reset_index() demotes the index to a column, or discards it with drop=True.
  • rename_axis names the index; rename(index=...) changes labels.
  • reindex forces an exact set and order of labels, filling gaps instead of raising.
  • Arithmetic aligns on labels, so unmatched labels produce NaN.
  • Duplicate labels are allowed but change loc results; check index.is_unique when a key must be unique.
  • Use the index for lookups, dates and alignment; keep ordinary data in columns.

Next up: indexes with several levels in MultiIndex.