Python Reference ยท Third-party Libraries
The pandas library
Overview of pandas: Series and DataFrame, selecting with loc and iloc, filtering, groupby, merge, missing values, and the functions you use most, with examples.
pandas is the standard Python library for tabular data. Its two main objects are the Series (one labeled column) and the DataFrame (a table of columns that share a row index). It reads and writes CSV, JSON, Excel and SQL, and makes filtering, grouping, joining and cleaning data short and readable.
Install it with python -m pip install pandas. It builds on NumPy, so many ideas (vectorized operations, dtypes, boolean masks) carry over. The examples run in the browser.
import pandas as pd df = pd.DataFrame({"name": ["Ada", "Bo"], "age": [36, 28]}) print(df["age"].mean())
Try it
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.
Core ideas
Index and labels. Every row has a label in the index. df.loc[label, column] selects by label (and slices include the end), df.iloc[row, col] selects by integer position (and slices exclude the end). Mixing these up is the most common beginner bug.
Boolean filtering. df[df["age"] > 30] keeps matching rows. Combine conditions with &, | and ~, and put each condition in parentheses.
Split, apply, combine. df.groupby("key") splits rows into groups; follow it with an aggregation such as .sum(), .mean() or .agg(...).
Missing data. Missing values appear as NaN (or pd.NA for the nullable types). Most aggregations skip them. Use isna(), dropna() and fillna() to find, remove or replace them.
Most used functions
| Function | Purpose |
|---|---|
pd.read_csv(path), df.to_csv(path) |
Read and write CSV files. |
pd.DataFrame(data) |
Build a table from a dict, list of dicts or array. |
df.head(), df.info(), df.describe() |
Quick looks at the data. |
df.loc[...], df.iloc[...] |
Select by label or position. |
df.sort_values(by), df.rename(columns=...) |
Reorder and relabel. |
df.groupby(by).agg(...) |
Grouped statistics. |
df.merge(other, on=..., how=...) |
Database-style joins. |
pd.concat([df1, df2]) |
Stack tables. |
df.pivot_table(...) |
Reshape into a summary table. |
df.apply(func) |
Run a function per row or column (slower than vectorized code). |
pd.to_datetime(series) |
Parse dates. |
Usage notes
- Most methods return a new object.
df.sort_values("x")does not changedfunless you assign the result. Avoidinplace=True; it rarely saves memory and is discouraged. - Chained indexing warns.
df[df.a > 1]["b"] = 0may not changedf. Writedf.loc[df.a > 1, "b"] = 0. With Copy-on-Write (the default in pandas 3), the chained form never changes the original. - Vectorize instead of looping. Column arithmetic and
.str/.dtaccessors are much faster thaniterrows(). - Dtypes matter. Check
df.dtypes. A column with one stray text value becomesobject, and numeric methods stop working. Convert withpd.to_numeric(col, errors="coerce"). - Merging can multiply rows when keys are not unique on one side. Check
len()before and after, or usevalidate="one_to_one". - Large files: read in chunks with
chunksize=, select columns withusecols=and use categories for repeated strings. - Printing and wrapping: wide tables are truncated by default;
df.to_string()prints everything.
Related
- NumPy, Matplotlib, SQLAlchemy,
csv - Lessons: NumPy, Machine learning basics, CSV