Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 64 ยท Getting Started

DataFrame Basics

Create a pandas DataFrame from dicts, records and arrays, then inspect it with shape, dtypes, head, tail, info and describe. Includes live examples.

The DataFrame is the object you will use most in pandas. It is a two-dimensional table with labeled rows (the index) and labeled columns. Each column is a Series, and different columns may have different data types: names as text, prices as decimals, dates as timestamps.

In this lesson you will build DataFrames in the common ways, learn the attributes that describe their structure, and use the handful of inspection methods that you run on nearly every new dataset.

flowchart TB
    subgraph DataFrame
      direction LR
      IDX["index<br/>0, 1, 2"]
      C1["column city<br/>(str)"]
      C2["column sales<br/>(int64)"]
      C3["column rate<br/>(float64)"]
    end

A DataFrame is an index plus named columns, each with its own dtype.

Building a DataFrame

From a dictionary of columns

This is the most readable form: each key is a column name and each value is a list (or Series or array) of that column's data. All columns must have the same length.

Python Dict of columns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

From a list of records

When your data arrives as one dictionary per row, which is typical of JSON from an API, pass the list directly. Keys become columns, and a missing key becomes a missing value:

Python List of records

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ben has no city, so that cell holds a missing value. Counting missing values per column is one line: df.isna().sum(). Missing Data covers what to do about them.

From a NumPy array or nested lists

With a 2-D array you supply the column names yourself, and optionally the index:

Python From an array with labels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rows, columns and the index

Three attributes describe the labels and the size of the table:

  • df.columns: the column labels.
  • df.index: the row labels (a RangeIndex by default, counting from 0).
  • df.shape: a tuple (number of rows, number of columns).

You can also use len(df) for the number of rows. To get one column as a Series, write df["sales"]; more selection tools come in Selecting Columns and Rows.

Python Structure of a DataFrame

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

dtypes: what type is each column?

df.dtypes lists one type per column. Checking it early prevents many surprises, such as numbers that were read as text and therefore refuse to add up.

Python Checking dtypes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In pandas 3, text columns show up as str. In older versions you would have seen object here.

Taking a first look

For a table you have not seen before, these four calls tell you almost everything:

Call What it shows
df.head(n) / df.tail(n) First or last n rows (default 5)
df.info() Columns, non-null counts, dtypes and memory use
df.describe() Summary statistics for numeric columns
df.sample(n) n random rows (use random_state= for repeatable output)

Let's generate a table of 200 fake orders with a fixed seed and inspect it:

Python head, tail and info

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

info() prints directly (it returns None), so do not wrap it in print. Its most useful column is Non-Null Count: if it is lower than the number of rows, that column has missing values.

describe()

describe() summarizes numeric columns with the count, mean, standard deviation, minimum, quartiles and maximum. Pass include="all" to cover text columns too:

Python describe()

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The 50% row is the median. For text columns, describe reports how many distinct values exist (unique), the most common one (top) and how often it occurs (freq); statistics that do not apply show NaN.

Printing wide tables

pandas shortens large tables when printing: it shows the first and last rows and hides the middle with .... To see everything as plain text, use to_string(); to limit what is shown, pass max_rows or max_cols:

Python to_string()

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first print of the 100-row table is abbreviated, while to_string(index=False) prints just the first four rows without the index column. More display options are in Styling and Displaying Tables.

Tip: inspect before you analyze. Run shape, dtypes, head() and describe() as the first thing you do with new data. Five seconds there save an hour of confusion later.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does df.shape return for a table with 5 rows and 3 columns?

    shape is a tuple of (rows, columns), so a table with 5 rows and 3 columns has shape (5, 3).
  2. 2. Which input creates one DataFrame row per dictionary?

    A list of dictionaries is treated as a list of records, where each dictionary becomes a row and the keys become columns.
  3. 3. Why should you not write print(df.info())?

    info() writes its summary directly to the output and returns None. Call df.info() on its own line.
  4. 4. Which method gives count, mean, standard deviation and quartiles of numeric columns?

    describe() computes summary statistics. info() reports structure, head() shows rows and dtypes lists column types.
  5. 5. Which attribute tells you whether a column was read as text or numbers?

    dtypes shows one data type per column, which is the first thing to check when numbers behave like text.

Summary

  • A DataFrame is a table of labeled columns (each a Series with its own dtype) and a row index.
  • Build one from a dict of columns, a list of record dicts, or a NumPy array with columns= and index=.
  • shape, columns, index, len(df) and dtypes describe its structure.
  • head(), tail(), info(), describe() and sample() give a quick first look; info() prints on its own.
  • Use to_string() to print a whole small table without abbreviation.

Next: Reading CSV Data, where tables come from text files instead of code.