Skip to content
Tutorials Center
My learning

Topics

Lesson 54 of 63 ยท Practice and Production

OCR and Text from Documents

Learn the OCR-to-NLP pipeline: error types, character error rate, cleaning hyphenated scans, and correcting OCR noise with confusion-weighted edit distance.

A great deal of the world's text is not text at all. It is a photo of a receipt, a scanned contract, a PDF made from a fax, a page of a book or a screenshot. Before any of the tools in this course can run, those pixels must be turned into characters. This step is optical character recognition (OCR), and it is the first stage of every document-processing pipeline: invoice extraction, archive search, ID checks, accessibility tools and digitized libraries.

OCR is never perfect, and its mistakes are not like typing mistakes. A person mistypes neighboring keys, but an OCR engine confuses letters that look alike: l, 1 and I, or rn and m. If you feed that text straight into tokenization, search or an entity recognizer, the errors multiply. In this lesson you will see where OCR fits in the pipeline, measure its errors with the character error rate, clean the typical layout damage of scanned text, and build a small post-correction step that uses a vocabulary and a confusion-aware edit distance. How an image gets prepared for OCR (grayscale, thresholding, deskewing) is covered in the OpenCV lesson on OCR; this lesson starts where the engine hands you a string.

The pipeline from page to meaning

flowchart LR
    A["Scan or photo"] --> B["Image cleanup: deskew, threshold"]
    B --> C["Layout analysis: blocks, lines, tables"]
    C --> D["Text recognition"]
    D --> E["Text cleanup and correction"]
    E --> F["NLP: entities, search, classification"]

Most quality problems appear early, but they are only noticed late. Fix them as close to their source as you can.

Image cleanup is where OpenCV helps. Layout analysis finds text blocks, columns, headers and tables, and decides the reading order, which is surprisingly hard: a two-column page read straight across mixes the columns. Recognition turns each line image into characters. Classic engines such as Tesseract segment and classify characters and use a small language model, and modern neural engines read whole lines with a CNN or Transformer plus CTC or an attention decoder, exactly as in speech recognition. The newest approach asks a large multimodal model to read the page directly and return structured text. All of them give you strings that are probably right, so the last two steps in the pipeline are yours.

What kinds of OCR errors are there?

Typical OCR errors

Look-alike swaps O and 0, l and 1, S and 5.
Merges and splits rn read as m, or a word cut in two.
Lost characters Faint strokes and dots vanish.
Layout damage Hyphens, columns, headers, footers.

Each type needs a different fix.

1 / 2

The first group is substitutions between similar shapes. The second is merges and splits: rn becomes m (so modern is read as modem) or m becomes rn, and in the other direction a wide gap can split one word into two. The third is deletions and insertions: thin strokes disappear, specks of dust become commas. The fourth is not about characters at all: it is layout damage, such as end-of-line hyphens, running page headers and broken reading order.

Measuring the damage: CER and WER

The standard metrics are the same as for speech recognition: the character error rate (CER) is the character-level edit distance between the OCR text and a hand-checked ground truth, divided by the number of characters in the ground truth, and the word error rate (WER) is the same at word level. See speech recognition for the full derivation. For OCR, CER is the more useful figure, because a single wrong character ruins a whole word, and the WER is therefore much larger. The example below keeps the edit operations, so you can also see which confusions occur, which tells you what to fix.

Python CER, WER and a count of the confusions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Eleven character errors in 82 characters is a CER of 13.4%, which sounds moderate. But nine of the fourteen words are wrong, a WER of 64.3%, and the wrong words include every number in the line: the invoice number, the day, the year, the day count and the amount. For a task like invoice extraction, OCR with a CER of 13% is useless without correction. The last line of the output shows the pattern: five of the eleven errors are the same confusion, zero read as the letter O, and the others are l for 1 or I, i for l, and l inserted before i in Marcli (the engine read h as li). A pattern like this is good news, because you can fix it systematically with rules.

Cleaning layout damage

Before you correct characters, repair the structure. Text from scanned books and PDFs typically has these problems:

  • Ligatures such as ๏ฌ (one character) are really fi. Unicode normalization (NFKC, see text cleaning) expands them.
  • Hyphenation at line ends: Reve- followed by nue must become Revenue.
  • Hard line breaks inside paragraphs: a single newline is only the page width, while a blank line separates paragraphs.
  • Running headers and footers such as Annual Report 2024 Page 3, repeated on every page, which would otherwise pollute word counts and search results.
  • Irregular spacing from justified text.
Python Cleaning a scanned page with regular expressions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two paragraphs come out as clean prose: the ligatures are gone, Revenue, European and improved are whole again, the headers have disappeared, and the spacing is even. The last line shows the unavoidable risk of the hyphen rule. well-known becomes wellknown and state-of-the-art becomes stateof-the-art, because the hyphen was real. Better systems check the joined word against a dictionary and keep the hyphen if only the hyphenated form exists. The header pattern is specific to this report, so in practice you find running headers by looking for lines that repeat on many pages.

Correcting characters with a vocabulary

For words, a classic recipe uses a vocabulary: if a token is not a known word, find the nearest known word and replace it, as in spelling correction. OCR gives that recipe a twist: errors are not uniform. Replacing l with i should cost less than replacing l with z, and reading rn as m is a single, very common mistake. A weighted edit distance gives common confusions a smaller cost, so the correction prefers the substitutions an OCR engine really makes. The version below uses a cost of 0.4 for known confusions and 1.0 for everything else, and accepts a replacement only if the weighted distance is at most 1.

Python Confusion-weighted edit distance for OCR correction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All seven damaged words are repaired: tlie becomes the, sliows becomes shows, tliat becomes that, cornpany becomes company, tbe becomes the and rnodern becomes modern, so the sentence goes from 10 of 17 correct words to 17 of 17. The multi-character rules do the work here: li for h and rn for m are cheap, so they beat unrelated words. The last line lists the limits, and they matter more than the success. Acme and Lisbon stay untouched, but only by luck: no vocabulary word happens to be within a weighted distance of 1, and a name that did look like a common word would be "corrected" into it. modem, a real word if you are writing about networks, is "corrected" to modern because the vocabulary does not contain it. And 0ffice is not repaired at all because the function refuses to touch tokens with digits. In real use, build the vocabulary from your domain, protect names, codes and numbers with regular-expression rules, and consider character-aware or language-model correction, which uses the surrounding words to choose among candidates.

Be careful with automatic correction on fields where a wrong guess is worse than a flagged one: amounts, IDs, dates and names. Fix known format errors (a digit-only field containing O can safely become 0), but route the rest to review instead of guessing.

Confidence scores and real engines

Engines report a confidence for each word, and you should use it. Low-confidence words go to correction or human review, high-confidence ones pass. The pytesseract call below is read-only, because it needs the Tesseract program and an image file:

Python
import pytesseract
from PIL import Image

image = Image.open("invoice.png")
data = pytesseract.image_to_data(image, output_type=pytesseract.Output.DICT, config="--psm 6")

words = []
for text, conf in zip(data["text"], data["conf"]):
    if text.strip():
        words.append((text, float(conf)))

uncertain = [w for w, c in words if c < 60]
print("words to review:", uncertain)
print(" ".join(w for w, _ in words))

For hard material (handwriting, receipts, tables, forms) neural document models and multimodal LLMs read layout and text together and can return structured JSON (vendor, date, line items), which often skips the cleanup steps above. They can still make confident mistakes, including inventing plausible text for a blurry number, so validate their output with checks you can compute: totals that add up, dates that parse, IDs that match a pattern.

Finally, measure on your own pages. Build a small, hand-corrected set of 30 to 50 pages across your worst sources (fax, phone photos, old prints), report CER and the field-level accuracy that your application cares about, and re-run the numbers every time you change the OCR engine or the correction step. See evaluation and responsible NLP and NLP in production for monitoring.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is OCR output different from typing errors?

    Engines confuse look-alike shapes, so corrections should favor those confusions.
  2. 2. In the invoice example the CER was 13.4% but the WER was 64.3%. What explains the gap?

    CER counts individual characters. WER counts a word as wrong when it contains any error, so scattered errors damage many words.
  3. 3. Why can the rule that joins "word-" + newline + "continuation" be harmful?

    At a line break, a hyphen can be either a split marker or part of the word. A dictionary check on the joined form helps.
  4. 4. What does a weighted edit distance change compared with the plain one?

    Cheaper known confusions make the correction choose the vocabulary word that an OCR engine would most plausibly have misread.
  5. 5. When should an OCR pipeline send a field to human review instead of auto-correcting it?

    A wrong auto-correction in a critical field is silent. Review catches uncertain values before they enter downstream systems.

Summary

  • OCR is the first step of document NLP: image cleanup, layout analysis, recognition, then text cleanup and correction before search, extraction or classification.
  • OCR errors are shape-based: look-alike swaps, merges and splits, lost characters, and layout damage such as hyphens and headers.
  • CER is character-level edit distance over ground-truth length; WER is much higher because one wrong character spoils a word. Count the confusions to learn what to fix.
  • Clean structure first (Unicode normalization, de-hyphenation, paragraph joining, header removal), accepting that rules sometimes damage real hyphens.
  • Correct words with a domain vocabulary and a confusion-weighted edit distance, protect names, numbers and codes, use confidence scores, and send risky fields to human review.

Next, see how language technology talks back in Chatbots and Dialogue Systems.