Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 64 ยท Data Types

Data Types and Conversion

Learn pandas data types: int, float, bool, str, category and datetime. Convert columns with astype, to_numeric and convert_dtypes, and use nullable integers.

Every column in a pandas DataFrame has exactly one dtype, the type that all of its values share. The dtype decides what a column can hold, which operations work on it, how much memory it uses and how fast calculations run. A column of numbers stored as text will happily print, but "10" + "5" gives "105" instead of 15, and sorting puts "9" after "10".

Most real data arrives with the wrong dtypes, because CSV files and web forms are just text. This lesson shows the dtypes you will meet, how to inspect them, and how to convert columns safely. If you have not met dtypes yet, DataFrame Basics shows where it appears.

The everyday dtypes

Here are the dtypes you will use most, and the Python values they hold:

dtype Holds Example
int64 whole numbers 3, -12
float64 decimal numbers (and NaN) 2.5, nan
bool True / False True
str text "pandas"
category a small set of repeated labels "small", "large"
datetime64[us] dates and times 2024-03-01 09:30
object any Python objects lists, dicts, mixed values
Python Inspect the dtypes of a DataFrame

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

df.dtypes lists one dtype per column. select_dtypes filters columns by type, which is handy when you want "all numeric columns" without naming them.

The default text dtype in pandas 3

In pandas 2 and earlier, text columns were stored as object, a catch-all that holds any Python object. pandas 3 infers a dedicated string dtype, shown as str, whenever a column holds only text (and missing values). It is faster, uses less memory and refuses to hold non-text values by accident.

Python Text columns get the str dtype

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Use object only for genuinely mixed data. If a column that should be text comes out as object, something non-text is hiding in it; finding that value is the first cleaning step.

Converting with astype

astype returns a new Series (or DataFrame) converted to the dtype you name. Pass a dict to convert several columns at once.

Python astype on strings read from CSV

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

read_csv already turned true/false into bool and the numbers into numeric dtypes. Sorting ["10", "5", "42"] as text gives ['10', '42', '5'], which is almost never what you want; after astype(int) it sorts numerically.

astype is strict. It raises an error rather than guess, which is the right default:

Python
pd.Series(["1", "x"]).astype(int)      # ValueError: cannot parse "x"
pd.Series([1.5, 2.0]).astype("Int64")  # TypeError: would lose the .5

For messy input you want a gentler tool.

to_numeric for messy numbers

pd.to_numeric converts text to numbers and lets you decide what happens to values that do not parse. The errors argument has two useful settings: "raise" (the default) stops with an error, and "coerce" turns bad values into missing values (NaN) so you can inspect them.

Python Coercing bad values with to_numeric

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

" 30 " parsed because surrounding spaces are tolerated, but "1,200" did not, so it became NaN until we stripped the comma. The pattern is always the same: coerce, then look at what failed. Silent coercion without that second look hides data problems. Working with Text covers the string methods used to clean input first.

Because NaN is a float, a column of integers with one gap becomes float64. That surprises people, and the next section fixes it.

Nullable integers and booleans

The classic NumPy dtypes int64 and bool cannot hold a missing value, so pandas upgrades an integer column to float as soon as one value is missing. The nullable dtypes (capitalised Int64, Float64, boolean) keep their type and mark gaps with pd.NA.

flowchart LR
    A["[1, 2, missing]"] --> B["int64?<br/>no room for a gap"]
    B --> C["float64<br/>1.0, 2.0, NaN"]
    A --> D["Int64<br/>1, 2, NA"]

A gap forces the classic integer dtype to become float; the nullable Int64 keeps integers and stores NA.

Python int64 versus Int64 with a missing value

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The classic dtype shows 1.0, 2.0, nan, the nullable one 1, 2, <NA>. Use nullable dtypes when whole numbers (ids, counts, years) must stay whole. The next lesson, Missing Data, compares NaN, NA and None in detail.

Letting pandas pick: convert_dtypes

convert_dtypes() looks at each column and chooses the best nullable dtype for it: whole-valued floats become Int64, text becomes string, booleans become boolean. It is a quick way to tidy a freshly loaded table.

Python convert_dtypes in one call

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that qty changed from float64 to Int64 because every non-missing value was a whole number. ratio stays Float64 since its values have fractions. Review the result; automatic choices are convenient, but you should know what you got.

Dates and categories

Two dtypes deserve a first look here; each has its own lesson.

  • Dates: text like "2024-03-01" should become a real datetime with pd.to_datetime. Then you can sort, subtract and group by month. See Dates and Times. In pandas 3 the resolution of a parsed datetime depends on the input, so it may show as datetime64[us] rather than the old datetime64[ns].
  • Categories: a column with few distinct values repeated many times (sizes, countries, statuses) can use category to save memory and gain a defined order. See Categorical Data.
Python Dates and categories from text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing the right dtype

A short checklist when a table loads:

  1. Print df.dtypes and compare it with what each column should be.
  2. Convert numbers stored as text with to_numeric(..., errors="coerce") and inspect the failures.
  3. Convert dates with to_datetime.
  4. Use category for low-cardinality labels, Int64 for whole numbers that may be missing.
  5. Leave object only for genuinely mixed values.

Smaller dtypes also save memory; Memory Usage and Smaller Dtypes measures how much.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What dtype does pandas 3 give to a Series created from ['a', 'b', 'c']?

    pandas 3 infers the dedicated str dtype for text columns. In older versions this was object.
  2. 2. Why does pd.Series([1, 2, None]) have dtype float64?

    The classic int64 dtype has no way to represent a gap, so pandas upgrades to float64 and uses NaN. Pass dtype='Int64' to keep integers and use NA.
  3. 3. What does pd.to_numeric(s, errors='coerce') do with a value that is not a number?

    coerce converts unparseable values to NaN so the rest of the column still converts. Inspect the NaN rows afterwards to see what failed.
  4. 4. Which call converts every column to the best-fitting nullable dtype automatically?

    convert_dtypes picks Int64, Float64, string or boolean where it can. astype needs an explicit dtype, and DataFrame has no to_numeric method.

Summary

  • Each column has one dtype; check it with df.dtypes before analysing.
  • pandas 3 stores text as str; object now signals mixed or non-text values.
  • astype converts strictly and raises on bad values; to_numeric(errors="coerce") turns failures into NaN for inspection.
  • Nullable dtypes (Int64, Float64, boolean) keep whole numbers whole and use pd.NA for gaps.
  • convert_dtypes() chooses sensible nullable dtypes automatically.
  • Use datetime for dates and category for repeated labels.

Next: Missing Data.