Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 63 ยท Getting Started with NLP

Working with Text in Python for NLP

The Python text toolkit every NLP project uses: string methods, regular expression essentials, Counter for word frequencies, sorting by count and f-string reports.

Before libraries like scikit-learn or spaCy, almost every NLP program is built from four everyday Python tools: string methods, regular expressions, collections.Counter and f-strings. They are enough to clean text, split it into words, count them and print a readable report. This lesson revisits those tools with an NLP mindset, so that the rest of the course can move quickly.

If any of this feels unfamiliar, the full explanations are in Python strings, string methods, regular expressions, collections and string formatting. Here we focus on the parts you will use again and again.

String methods you will use constantly

Python strings are immutable sequences of characters, and every method returns a new string. The ones that matter most for text work:

  • lower() and casefold() change case; casefold() is the stronger, language-aware version meant for comparisons (it maps German รŸ to ss).
  • strip(), lstrip() and rstrip() remove whitespace or the characters you list from the ends.
  • split() breaks on whitespace by default; split(",") breaks on a separator; "-".join(parts) glues pieces back together.
  • replace(old, new) swaps substrings; startswith() and endswith() test the edges.
  • count(sub) counts non-overlapping occurrences and find(sub) returns the first index, or -1.
Python String methods on a paragraph

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two details matter. split(",") keeps the empty string between two commas, whereas split() with no argument collapses runs of whitespace and never returns empty strings. And casefold() treats STRASSE and straรŸe as equal, which plain lower() does not.

Regular expressions in five patterns

Plain methods find fixed strings. A regular expression (regex) describes a pattern, and Python's re module applies it. You need only a small part of regex syntax for NLP:

Pattern Meaning
\d, \w, \s a digit, a word character, whitespace
[a-z'] any one character from a set
+, *, ? one or more, zero or more, optional
exactly four of the previous item
\b a word boundary
( ) a group you can pull out later

The functions you will call most are re.findall (all matches as a list), re.sub (replace matches), re.search (the first match, as a match object) and re.fullmatch (the whole string must match). Write patterns as raw strings (r"...") so backslashes reach re unchanged.

Python Five regex patterns for text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line is the pattern the course uses as a first tokenizer: any run of lowercase letters and apostrophes. It keeps don't in one piece and throws away punctuation. Tokenization refines it, and Rule-Based Information Extraction uses regexes for emails, dates and prices.

Regexes are powerful but easy to get subtly wrong. A pattern that works on your five test strings can fail on real data. Test with tricky inputs, prefer simple patterns, and remember that regex alone cannot understand grammar or meaning.

Counting with Counter

Counting words is the foundation of classic NLP. A Counter is a dictionary specialized for counting: give it any iterable and it returns a mapping from item to how often it appeared. Its most_common(n) method returns the top n pairs, sorted by count. Missing keys count as zero instead of raising KeyError.

Two vocabulary words from linguistics are worth learning here. A token is one occurrence of a word in the text, and a type is a distinct word. "the cat saw the dog" has five tokens and four types. The ratio of types to tokens (the type-token ratio) is a rough measure of vocabulary richness.

Python Counting and ranking words

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

There are 24 tokens but only 20 types. The word text leads with 3 occurrences, then messy and words with 2 each. Sorting by (-count, word) is a common idiom: the minus sign sorts counts from high to low, and the second key makes ties deterministic, which matters when you need reproducible output.

Formatting results with f-strings

NLP experiments produce a lot of numbers, and readable output saves time. f-strings take a format spec after a colon: {value:.1%} prints a percentage with one decimal, {n:,} adds thousands separators, and {word:<10}, {n:>3} and {x:^5} left-align, right-align and center in a fixed width. The one-line '#' * n trick turns a count into a tiny bar chart, handy for quick looks at a distribution.

Python Format specs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A mental model for the rest of the course

Notice the shape of what we just did: raw string, then tokens, then counts, then a report. Almost every lesson that follows is a variation. Cleaning changes the first step, tokenizers change the second, TF-IDF and embeddings change the third, and classifiers turn the counts into predictions. If you keep that shape in mind, new techniques will feel like small upgrades, not new subjects.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does 'a,b,,c'.split(',') return?

    split with an explicit separator keeps empty fields. Only split() with no argument collapses whitespace and drops empty strings.
  2. 2. Why write regex patterns as raw strings such as r'\d+'?

    In a normal string, sequences like \n or \b are interpreted by Python first. A raw string passes the backslashes through to the regex engine.
  3. 3. The text 'the cat saw the dog' has how many tokens and types?

    Every occurrence is a token (five), while the repeated word 'the' counts once as a type (four distinct words).
  4. 4. Why sort with key=lambda kv: (-kv[1], kv[0]) when printing word counts?

    The negated count sorts descending, and the word as second key makes the order of equal counts deterministic.

Summary

  • Core string methods (lower, casefold, strip, split, join, replace, count, find) cover most basic cleaning; casefold is the right choice for caseless comparison.
  • re offers findall, sub, search and fullmatch; use raw strings and test patterns on tricky inputs.
  • A token is one occurrence of a word and a type is a distinct word; Counter and most_common give word frequencies in one line.
  • Sort by (-count, word) for stable, reproducible rankings.
  • f-string format specs turn numbers into readable reports; the raw text, tokens, counts, report shape returns throughout the course.

Next: The NLP Toolkit: NLTK, spaCy, scikit-learn and Transformers