Skip to content
Tutorials Center
My learning

Topics

Lesson 134 of 149 ยท Projects

Project: Text Analyzer

Build a text analyzer in Python step by step. Count words and sentences, find the most common words with Counter, estimate readability and print a clean report.

Word processors count your words, writing tools grade your readability, and search engines rank pages by the words they contain. All of them start with the same job: turning a block of text into numbers. In this project you'll build your own text analyzer that reads any text and prints a report with word counts, the most frequent words, sentence statistics, a readability score and a small bar chart.

Along the way you'll combine string methods, regular expressions, collections.Counter, sets, functions, data classes and f-string formatting. The program grows in six stages; each one is complete and runnable, so you can watch the analyzer get smarter step by step.

The sample text

Every stage analyzes the same short passage, stored in a triple-quoted multi-line string called TEXT. Feel free to replace it with anything you like: a paragraph from your own writing, a song you wrote, or a few lines of a story.

Stage 1: first counts

Start with numbers you can get from basic string tools. len() counts characters, str.splitlines() splits into lines, and str.split() with no arguments splits on any run of whitespace:

Python Stage 1: characters, lines and words

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the last line of output. split() only cuts at whitespace, so punctuation sticks to the words: city. and tomatoes, are counted as different words from city and tomatoes. For a real analysis we need cleaner words.

Stage 2: clean words with a regular expression

A regular expression can pull out exactly the pieces we want. The pattern [a-z]+(?:'[a-z]+)? means "one or more letters, optionally followed by an apostrophe and more letters", so it keeps contractions like don't in one piece. Lowercasing the text first makes The and the the same word:

Python Stage 2: clean words

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A few things to notice:

  • re.compile() builds the pattern once and gives it a name. That's both faster and more readable when a pattern is reused.
  • Converting the list to a set removes duplicates, so len(set(words)) is the number of different words. The vocabulary ratio (unique words divided by all words) hints at how varied the writing is.
  • sorted(..., key=len, reverse=True) sorts by length, longest first; see Higher-Order Functions.
  • The test line at the end shows the pattern's limits: numbers vanish and mice-and-men becomes three words. Decisions like these are part of every text-processing job; there is no single "correct" definition of a word.

Stage 3: the most common words

Counting how often each word appears is a job made for Counter. Give it a list and it builds a dictionary of counts; most_common(n) returns the top n pairs. The raw winners, though, are always small words like "the" and "and". We filter those stop words out with a set:

Python Stage 3: word frequencies

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Checking word not in STOP_WORDS is fast because sets look up items in constant time, no matter how big they get. Real projects use stop-word lists with a few hundred entries; ours is just big enough for the sample.

Stage 4: sentences

Sentences end with ., ! or ?. Splitting right after one of those characters, where whitespace follows, gives us the sentences. The pattern (?<=[.!?])\s+ uses a lookbehind (?<=...), which checks the previous character without removing it, so each sentence keeps its punctuation:

Python Stage 4: sentence statistics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sample has five lines but seven sentences, because some lines hold two. sum(s.endswith('?') for s in sentences) counts questions in one line: True counts as 1 and False as 0 when you add them up.

This simple rule does get fooled by abbreviations such as "Dr. Lee" or "e.g. this". Handling every case needs a language-processing library; for our analyzer the simple version is good enough.

Stage 5: a readability score

How hard is a text to read? A long-standing answer is the Flesch reading-ease score. It is computed from just two ratios: average words per sentence and average syllables per word. Longer sentences and longer words mean a lower score:

TEXT
score = 206.835 - 1.015 ร— (words / sentences) - 84.6 ร— (syllables / words)

Scores around 60โ€“70 are plain English; above 80 is very easy, and below 30 reads like an academic paper. Counting syllables exactly needs a dictionary, but a rough estimate works well: count groups of vowels, and don't count a silent e at the end of a word:

Python Stage 5: syllables and readability

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The estimate isn't perfect ("idea" has three syllables, but the rule counts two because "ea" forms a single vowel group), yet over hundreds of words the errors largely cancel out. reading_ease(*counts) uses argument unpacking to pass the tuple's three numbers as three arguments.

Stage 6: the complete analyzer

Now we assemble everything. The finished program:

  • collects every statistic into a data class called TextStats, so the analysis and the printing are separate,
  • prints a formatted report with a bar chart of the top words,
  • and analyzes your own text if you type it in the Input box, falling back to the sample when the box is empty.

Reading "all the lines in the Input box" uses a loop that calls input() until it raises EOFError, which is how input() signals that there's nothing left to read. The preset input below is a short original text; clear the box to analyze the built-in sample instead:

Python Stage 6: the complete text analyzer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Some details worth a second look:

  • @property on avg_sentence_length and reading_ease computes those values from the stored fields, so they can never get out of sync (see Properties).
  • max(len(split_sentences(text)), 1) prevents a division by zero for text without any sentence ending, and analyze() raises a ValueError for text without words.
  • f"{stats.characters:,}" adds thousands separators, and "TEXT REPORT".center(40) centers the title.
  • The bar chart scales every bar relative to the most frequent word, so the longest bar is always 20 characters wide.
  • read_all_input().strip() or SAMPLE uses the fact that an empty string is falsy: if nothing was typed, or falls back to the sample.

Analyze a real file. On your own computer, replace read_all_input() with Path("essay.txt").read_text(encoding="utf-8") from pathlib and point it at any text file.

Ideas to extend the analyzer

  1. Compare two texts. Analyze two passages and print their statistics side by side, with the difference in readability.
  2. Word pairs. Count the most common two-word phrases with zip(words, words[1:]) and a Counter.
  3. Find repeated words. Flag accidental doubles like "the the" with a regular expression using a backreference: r"\b(\w+)\s+\1\b".
  4. Export JSON. Save the stats with dataclasses.asdict() and the json module.
  5. Test it. Write unit tests for count_syllables() and split_sentences(), including tricky inputs.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does stage 1 count city. and city as different words?

    str.split() knows nothing about punctuation. A regular expression that matches only letters gives cleaner words.
  2. 2. What does Counter(words).most_common(3) return?

    most_common(n) returns a list of (element, count) tuples, from the most to the least frequent.
  3. 3. Why are the stop words stored in a set instead of a list?

    A set lookup takes about the same time no matter how big the set is, while in on a list checks items one by one.
  4. 4. How does read_all_input() know when to stop reading?

    EOFError ("end of file") is how input() reports that there is nothing more to read, so the loop catches it and returns the collected lines.

Summary

  • Basic string tools (len, split, splitlines) give quick counts but don't understand punctuation.
  • A compiled regular expression extracts clean words and splits sentences; lookbehind keeps the punctuation in place.
  • Counter and most_common() count frequencies in one line; a set of stop words filters out filler words quickly.
  • The Flesch reading-ease score turns words, sentences and estimated syllables into a single readability number.
  • Separating analysis (analyze() returning a data class) from presentation (print_report()) keeps the program easy to change and test.

Next: Project: Inventory Manager, where you'll model a shop's stock with classes and save it as JSON.