Skip to content
Tutorials Center
My learning

Topics

Lesson 110 of 149 ยท Data and the Web

Pandas Basics

Learn pandas step by step. Build Series and DataFrames, read CSV data, select and filter rows, add columns, sort, group with groupby and handle missing values.

NumPy gives you fast arrays of numbers, but real data usually looks like a spreadsheet: named columns of different types, rows that belong together, a few empty cells. pandas is the library for exactly that kind of data. Its main type, the DataFrame, is a table with labeled rows and columns, and it comes with tools for loading, cleaning, filtering, grouping and summarizing data in a few readable lines.

In this lesson you'll build Series and DataFrames, read CSV data, select and filter rows, add calculated columns, sort, group with groupby() and handle missing values. If you worked through the CSV lesson or the data report project, you'll see how much shorter the same kinds of tasks become.

Note: pandas is a third-party package (pip install pandas on your own computer). The editors here load it automatically on import pandas, but the first Run downloads pandas and NumPy, which is several megabytes, so expect a short wait. After that, runs start quickly.

Series: a labeled column

A Series is a one-dimensional array with a label for every value, called the index. Think of it as a cross between a list and a dictionary: values stay in order, but you can also look them up by label.

Python Creating a Series

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The usual alias is import pandas as pd. Each Series has a dtype like a NumPy array, and math on a Series runs element by element. The labels travel with the values, so a filtered Series still knows which month each number belongs to.

DataFrame: a table of columns

A DataFrame is a collection of Series that share one index. Each column has a name and its own dtype, so a text column can sit next to a number column. The quickest way to build one by hand is a dictionary of lists, one list per column:

Python Building a DataFrame

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The numbers on the left (0 to 3) are the default index. Columns are Series, so books["pages"] gives you a Series with all the Series tools.

Reading CSV data

In practice you rarely type data in by hand. pd.read_csv() loads a CSV file into a DataFrame, detects the header, and converts number columns to numbers automatically. It accepts a file path, a URL, or any file-like object. Here we use io.StringIO to treat a string as a file, so the example needs no real file:

Python read_csv() and a first look

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

head() and tail() show a few rows so you can check the data loaded correctly. describe() summarizes number columns: count, mean, standard deviation, minimum, quartiles and maximum. parse_dates=["date"] turns the date column into real timestamps instead of text, which you'll use in a moment. Another quick check is sales.info(), which lists each column with its dtype and the number of non-empty values.

Reading from a file on disk is the same call with a path: pd.read_csv("sales.csv"). pandas can also read Excel, JSON, SQL databases and more with functions such as read_excel(), read_json() and read_sql().

Selecting columns and rows

There are three main ways to select data, and it's worth learning the difference early:

  • df["col"] selects one column as a Series; df[["a", "b"]] (note the double brackets) selects several columns as a new DataFrame.
  • df.loc[rows, cols] selects by label: index labels and column names. Label slices include the end.
  • df.iloc[rows, cols] selects by position, like list indexing. Position slices exclude the end.
Python Columns, loc and iloc

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A selection with a single row or column comes back as a Series; anything wider stays a DataFrame. When you only need one cell, df.loc[row, col] or df.at[row, col] is the clearest way to write it.

Filtering rows

Filtering works like NumPy masks. A comparison on a column gives a Series of True/False values, and putting it inside df[...] keeps the matching rows:

Python Filtering with conditions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same rules as NumPy apply: combine conditions with &, | and ~, and wrap each one in parentheses. isin() checks membership in a list, and text columns have string methods under .str, such as sales["product"].str.startswith("L"). The query() method lets you write the condition as a string with plain and/or, which some people find easier to read.

Adding and changing columns

Assigning to a new column name adds a column. Because the math is vectorized, one line computes the value for every row:

Python Calculated columns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

  • Use vectorized math and .str methods when you can; they're fast and clear.
  • apply() runs a Python function on every value. It's flexible but slower, so save it for logic that has no vectorized form.
  • To change only some rows, use df.loc[mask, "column"] = value.
  • Methods like rename(), drop() and sort_values() return a new DataFrame and leave the original alone. Assign the result if you want to keep it.

Avoid chained assignment. Code like df[df["units"] > 2]["price"] = 0 looks reasonable but changes a temporary copy, not df. Recent pandas versions (with "copy-on-write") never let it modify the original. Always do it in one step: df.loc[df["units"] > 2, "price"] = 0.

Sorting

sort_values() sorts by one or more columns. Pass a list of columns to break ties, and ascending=False for largest first. nlargest() and nsmallest() are shortcuts for "top N" questions:

Python Sorting rows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

sort_index() sorts by the index labels instead. Like other DataFrame methods, sorting returns a new table, so write cities = cities.sort_values(...) to keep the order.

Grouping with groupby()

"Total sales per region" or "average score per class" are split-apply-combine questions: split the rows into groups, apply a calculation to each group, and combine the results. In pandas that's groupby():

Python groupby() and agg()

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

  • df.groupby("col")["other"].sum() gives one total per group, as a Series indexed by the group names.
  • agg() with named aggregations (new_name=("column", "function")) builds a summary table with several statistics at once. Common functions are "sum", "mean", "count", "min", "max" and "median".
  • You can group by a calculated Series too. Date columns have helpers under .dt, such as .dt.month, .dt.year and .dt.day_name().
  • value_counts() counts how often each value appears, like a Counter.

Add .reset_index() to a grouped result when you want the group names back as a regular column, for example before saving it.

Missing values

Real data has gaps. pandas marks missing values as NaN (for numbers) or NaT (for dates), and every method knows how to skip them. Find them with isna(), then decide whether to fill or drop them:

Python Finding and fixing missing data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Which fix is right depends on the meaning. A missing number of hours might really be 0, or it might be unknown, in which case filling it with 0 would pull the average down. fillna() and dropna() return new DataFrames; pass subset= to dropna() to look at specific columns only.

Tip: an integer column with a gap is stored as floats, because NaN is a float. That's why hours shows 12.0 instead of 12. After filling the gaps you can convert back with astype(int).

Saving results

to_csv() writes a DataFrame to a file. Without a path it returns the CSV text, which is handy for checking the output. Pass index=False unless the index carries real information:

Python Writing CSV

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

There are matching writers for other formats, such as to_json(), to_excel() and to_sql(). For a database workflow, the sqlite3 lesson shows how to store tables without any extra packages.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does df[["name", "age"]] return?

    Double brackets pass a list of column names, and the result is a DataFrame with those columns.
  2. 2. Which line keeps only the rows where price is over 100 and stock is above 0?

    Column comparisons give masks, which combine with &; each comparison needs its own parentheses.
  3. 3. What is the difference between df.loc and df.iloc?

    loc uses index labels and column names (and its slices include the end); iloc uses positions like a list.
  4. 4. Which expression gives the average salary for each department?

    groupby() splits the rows by department, then mean() runs on the salary column of each group.
  5. 5. After df.sort_values("score"), why is df still in its old order?

    Most DataFrame methods return a new object instead of changing the original, so assign the result.

Summary

  • pandas works with labeled tables. A Series is one labeled column; a DataFrame is a set of columns that share an index.
  • pd.read_csv() loads CSV data (from a path or a file-like object such as io.StringIO); head(), shape, describe() and info() give a first look.
  • Select columns with df["col"] or df[["a", "b"]], rows and cells with loc (labels) and iloc (positions).
  • Filter with masks combined by &, | and ~, or with isin() and query().
  • Add columns with vectorized math, change selected rows with df.loc[mask, "col"] = value, and never use chained assignment.
  • sort_values(), nlargest(), groupby() with agg() and value_counts() answer most summary questions.
  • Find gaps with isna(), fix them with fillna() or dropna(), and save results with to_csv(index=False).

Next: Plotting with Matplotlib.