Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 64 ยท Data Types

Working with Text

Clean text columns in pandas with the .str accessor: lower, strip, replace, split, extract with regex, contains, cat and padding, plus missing-value pitfalls.

Text columns are everywhere: names, addresses, product codes, free-form comments. They are also the messiest data you will meet, with stray spaces, inconsistent capitals and several values packed into one cell. pandas gives every text column a .str accessor, a toolbox of string methods that run on a whole column at once.

If you know Python's string methods like lower() and split(), you already know most of the names. The difference is that s.str.lower() applies the method to every element, skips over missing values instead of crashing, and returns a new Series. This lesson builds on Data Types and Conversion and prepares you for the cleaning work in Data Cleaning in Practice.

The .str accessor

The accessor only exists on text columns. Using it on numbers raises an AttributeError, which is a helpful signal that you forgot to convert something.

Python Cleaning case and whitespace

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the missing value (None) stays missing (nan) through the whole chain; it never causes an error. Also notice str.len() returns floats, because a column with a gap cannot hold plain integers (see Missing Data).

The most common cleaning methods:

Method What it does
lower(), upper(), title(), capitalize() change capitalisation
strip(), lstrip(), rstrip() remove surrounding whitespace (or listed characters)
replace(old, new, regex=...) replace text
len() number of characters
startswith(), endswith(), contains() boolean tests
slice(a, b) or s.str[a:b] take part of each string

Replace: plain text or regex

str.replace has a regex argument. Set it explicitly every time, because a dot, a plus or a bracket means something special in a regular expression.

Python replace with and without regex

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Regular expressions are a language of their own; Python's re module lesson (Regular Expressions) introduces the syntax, and pandas uses the same patterns.

Searching: contains, startswith and match

contains returns a boolean Series, ready to use as a filter (Filtering with Conditions). It treats the text as a regular expression by default, so pass regex=False for a literal search, and case=False to ignore capitals.

Python Filtering rows by text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

startswith and contains give False for the missing SKU when the column has the default str dtype, so the filter keeps working. Older object columns could return NaN there and break boolean filtering; if you hit that, pass na=False.

Splitting into pieces

str.split breaks each string at a separator. By default you get a column of lists; with expand=True you get a DataFrame with one column per piece, which is usually what you want. n= limits the number of splits.

Python split into columns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

str.cat concatenates: with one argument it joins the elements of a single Series into one string, and with another Series (or list) it joins them element by element. A missing value in either side makes the result missing unless you give na_rep=.

Extracting with regular expressions

str.extract pulls pieces out of each string using capture groups. Each group becomes a column, and rows that do not match get a missing value. Naming the groups names the columns.

flowchart LR
    A["INV-2024-001"] --> B["pattern<br/>INV-(year)-(num)"]
    B --> C["year: 2024"]
    B --> D["num: 001"]
    E["bad"] --> F["no match"] --> G["NaN, NaN"]

extract returns one column per capture group; a string that does not match produces missing values.

Python extract with named groups

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

extract returns the first match only. Use findall for all matches (a list per row), and fullmatch when the whole string must follow the pattern, which makes it a good validity check.

Padding and formatting

Identifiers often need a fixed width, such as zip codes that lost their leading zero when read as numbers.

Python Padding codes to a fixed width

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

zfill pads with zeros on the left, pad and center let you choose the filler, and get_dummies turns delimited tags into one 0/1 column per tag, which is useful for machine learning. Reshaping shows the more general pd.get_dummies.

More string methods: slice, join, partition, extractall and normalize

A few accessors are easy to miss. They are not needed every day, but each one replaces a clumsy workaround.

  • str.slice(start, stop, step) cuts every string like Python slicing. s.str.slice(0, 3) is the same as s.str[:3]. The method form helps in method chains, and str.slice_replace swaps a slice for new text.
  • str.join(sep) is for cells that already hold lists (for example after str.split). It glues each list into one string, so it is the reverse of split.
  • str.partition(sep) splits at the first separator into exactly three parts: before, the separator, and after. rpartition splits at the last one. Unlike split, the result always has three columns, which makes it safe for strings that sometimes lack the separator.
  • str.extractall(pattern) is extract for every match. It returns one row per match with a two-level index: the original row label and a counter called match.
  • str.normalize(form) applies Unicode normalisation ("NFC", "NFKC" and so on), so that text which looks the same but is stored differently (an accented letter written as one character or as a base letter plus a combining accent) compares as equal.
  • str.cat you met above; str.repeat repeats text, and str.wrap(width) inserts line breaks.
Python slice, join, partition, extractall and normalize

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

extractall is the tool for "find every number, hashtag or e-mail in a text". Name the groups, (?P<phone>\d{3}-\d{4}), to get named columns, and use groupby(level=0) to bring the results back to one value per original row. Always run normalize("NFC") before comparing or de-duplicating text that came from different systems (a Mac file name, a web form and a database can each use a different form).

Testing what a string contains: isalpha, isdigit and friends

A family of methods mirrors Python's str.isalpha() and friends. Each returns a Boolean for every row, so they are ideal for validation and for building filters: df[df["code"].str.isdigit()].

Method True when every character is...
isalpha a letter
isdigit a digit (includes superscripts such as ยฒ)
isdecimal a decimal digit, the strictest numeric test (only characters that can form a base-10 number)
isnumeric any numeric character (digits, fractions, roman numerals)
isalnum a letter or a digit
isspace whitespace
islower / isupper / istitle lowercase / uppercase / title-cased

An empty string is False for all of them, and a missing value is False too in a str column. Use isdecimal when you plan to convert with astype(int). Two more helpers: str.rfind(sub) returns the position of the last match (find returns the first), and -1 means not found, which makes it perfect for file extensions. str.swapcase() flips every letter's case, and str.casefold() lowercases aggressively for case-insensitive comparison of international text.

Python Character tests, rfind and swapcase

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note how "ยฒ" is a digit and numeric but not decimal, and how the Arabic-Indic digits ูฃูค pass all three numeric tests. To keep only plain ASCII digits, use str.fullmatch(r"[0-9]+").

Missing values and mixed types

With the str dtype, missing values flow through .str methods quietly: text methods return a missing value, and boolean tests such as contains return False. Two things still catch people out:

  • A column that is object because it mixes text and numbers: .str methods return NaN for the numeric entries, so values silently disappear from your result. Convert with astype(str) or clean the odd entries first.
  • Numbers stored as text: .str methods work, but arithmetic does not. Convert with pd.to_numeric when you are done cleaning.

Vectorised versus apply

You can call any Python function on each element with apply, so why bother with .str? Because the .str methods are shorter, handle missing values, and are generally faster. They give the same answer here:

Python .str versus apply

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reach for apply only when no .str method does the job, for example when the logic needs several columns or a custom function. apply, map and replace covers that case.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens when you call .str.lower() on a column of integers?

    The .str accessor only exists on text columns. Convert with astype(str) first if that is what you want.
  2. 2. Why should you write s.str.replace('.', ',', regex=False) when replacing a literal dot?

    With regex=True a dot is a wildcard, so every character would be replaced. regex=False treats the dot as plain text.
  3. 3. What does s.str.extract(r'(\d+)-(\d+)') return?

    extract returns a DataFrame with a column for each capture group (a Series if there is just one group and expand=False). Non-matching rows hold missing values.
  4. 4. Which call splits 'Ana Kim' into two columns?

    split alone returns lists. expand=True spreads the pieces into separate DataFrame columns.

Summary

  • The .str accessor applies string methods to a whole column and keeps missing values missing.
  • Clean with strip, lower/title, replace (always set regex=), and slice.
  • Search with contains, startswith, endswith, match and fullmatch; use the result as a filter.
  • split(expand=True) makes columns, extract pulls regex capture groups, findall returns all matches, cat joins.
  • zfill, pad and center fix widths; get_dummies expands delimited tags.
  • partition, join, extractall and normalize("NFC") handle first-separator splits, list cells, every regex match and Unicode comparisons.
  • Prefer .str methods over apply for text.

Next: Dates and Times.