Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 63 ยท Cleaning and Tokenization

Sentence Segmentation

Split text into sentences in Python. See why a period is not enough, build a rule-based splitter that handles abbreviations, and measure it with precision and recall.

Many NLP steps work one sentence at a time: translation, summarization, sentiment per sentence, and part-of-speech tagging all expect a list of sentences, not one giant string. Splitting text into sentences is called sentence segmentation (or sentence boundary detection), and it looks trivial: cut at every period, question mark and exclamation mark.

It is not trivial. A period can end a sentence, but it can also end an abbreviation (Dr.), sit inside a number (3.14), or mark an initial (J. K. Rowling). In this lesson you will see the problem fail, write a small rule-based splitter, and measure how good it is. You should already be comfortable with the tokenization basics and Python text handling.

Why the period is ambiguous

Punctuation like ? and ! is almost always a boundary. The period is the troublemaker, because it does at least four different jobs:

One dot, four jobs

Sentence end It rained all day.
Abbreviation Dr. Smith, Inc., etc.
Decimal point The ratio is 3.14.
Initial J. K. Rowling

Only the first one is a boundary.

1 / 2

Let us watch the naive approach fail. The code below marks every . , ? and ! as a boundary.

Python The naive splitter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The text contains only five real sentences, yet the naive splitter produces eight pieces. Dr., U.S. and Inc. each cut a sentence in the wrong place. Notice that it did correctly keep 3.5 together, because there is no space after that dot.

A rule-based splitter

The classic fix is a small decision procedure. First find every candidate boundary: sentence-ending punctuation followed by whitespace. Then decide, for each candidate, whether it is a real boundary by looking at the word before it and the word after it.

flowchart TD
    A["Candidate: punctuation + space"] --> B{"Is it ? or !"}
    B -- yes --> Y["Boundary"]
    B -- no --> C{"Word before is a title or initial?"}
    C -- yes --> N["No boundary"]
    C -- no --> D{"Word before is another abbreviation?"}
    D -- yes --> E{"Next word starts a sentence?"}
    E -- yes --> Y
    E -- no --> N
    D -- no --> F{"Next word starts uppercase or digit?"}
    F -- yes --> Y
    F -- no --> N

Titles never end a sentence; other abbreviations only do when the next word looks like a sentence opener.

Three small word lists carry the knowledge: titles that never end a sentence (Dr, Mr), other abbreviations that sometimes do (Inc, U.S), and common sentence starters (He, The) that help with the second group. Real systems learn such lists from data, but the logic is the same.

Python A splitter with abbreviation rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All seven sentences come out whole. The splitter keeps Dr. Smith, U.S. in 2019 and Acme Inc. now together because the next word is lowercase, and it still splits Inc. He when a sentence starter follows. The closing quote after Stop. is attached to the sentence it belongs to, thanks to the second capture group in the pattern.

Look at the order of the checks. Cheap, certain rules (titles, initials) run first, and the fuzzy rules run last. When you extend a splitter, add new rules at the position that matches their certainty.

Evaluating a splitter

"It looks right on my example" is not a measurement. To compare splitters fairly, write a small gold standard: text where you know the correct boundaries. Then compare the character offsets where each splitter ends a sentence. Every predicted boundary that is also in the gold set is a hit, giving the usual numbers:

  • Precision: of the boundaries predicted, how many were right?
  • Recall: of the true boundaries, how many were found?
  • F1: the harmonic mean of the two.

The naive splitter has plenty of false boundaries, which hurts precision. A splitter that never cuts has perfect precision on an empty set but zero recall. You need both numbers, which is why evaluating classifiers uses them too.

Python Precision, recall and F1 for boundaries

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The numbers are printed as (precision, recall, F1). The naive splitter finds every true boundary (recall 1.0) but invents five false ones, so precision drops to 0.5. The rule-based splitter scores 1.0 on all three numbers here. Do not trust that on such a tiny set: 9.30 a.m. on Monday survives only because the next word is lowercase, and a sentence that began with a.m. followed by a capitalized name would defeat the rules. A larger gold set exposes more of these cases. Collect them as you find them; they become your regression test.

When to use a library instead

Hand-written rules are a good teaching tool and fine for a controlled domain such as your own logs or product descriptions. For general text, use a trained splitter. The usual choices (shown read-only here, because they are not installed in the browser runner) are NLTK's Punkt model and spaCy's pipeline:

Python
# NLTK: an unsupervised model that learns abbreviations from a corpus
from nltk.tokenize import sent_tokenize
sent_tokenize("Dr. Smith arrived. He sat down.")

# spaCy: sentence boundaries come from the dependency parser
import spacy
nlp = spacy.load("en_core_web_sm")
[s.text for s in nlp("Dr. Smith arrived. He sat down.").sents]

Punkt treats the problem as statistics: it learns which tokens usually precede a period without ending a sentence. spaCy decides from grammar, which also handles text with no punctuation at all, such as transcripts. Both still make mistakes on messy input like chat logs, lists and tweets, where "sentence" itself is a fuzzy idea.

Practical tips

  • Normalize first. Collapse repeated whitespace and fix line-wrapped paragraphs before splitting (see cleaning text); a hard line break inside a sentence is not a boundary.
  • Keep offsets when you can. If you need to highlight sentences in the original text, store start and end positions instead of cutting strings.
  • Decide what a sentence is for lists and headings. Bullet lines and titles have no period at all; many pipelines treat each line as its own unit.
  • Test on your own domain. Legal, medical and scientific text are full of abbreviations (Fig., et al., Sec.) that general models only partly know.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is splitting on every period a poor sentence splitter?

    Periods end sentences, but they also appear in Dr., U.S., J. K. and 3.14, so a plain split creates false boundaries.
  2. 2. In the rule-based splitter, why are titles such as "Dr" checked before other abbreviations?

    A sentence does not end in "Dr.", so that rule needs no extra evidence. Fuzzier rules come later.
  3. 3. A splitter that never splits anything has which evaluation result on a text with several true boundaries?

    It finds none of the true boundaries, so recall is 0 and F1 is 0, no matter how "careful" it is.
  4. 4. Which is a reason to prefer a trained splitter such as Punkt over hand-written rules for general text?

    Punkt estimates from a corpus which tokens tend to precede a period without ending the sentence, which covers far more cases than a short list.

Summary

  • Sentence segmentation splits text into sentences; ? and ! are easy, the period is ambiguous.
  • Find candidate boundaries first, then accept or reject each using the word before and after.
  • Apply certain rules (titles, initials) before fuzzy ones (abbreviation plus sentence starter).
  • Evaluate with precision, recall and F1 on boundary offsets from a small gold standard.
  • Use Punkt or spaCy for general text, and always test on your own domain.

Next, remove the words that carry little meaning in Stop Words and When to Keep Them.