Skip to content
Tutorials Center
My learning

Topics

Lesson 156 of 171 ยท Projects

Project: Text Statistics

Build a C++ text analyzer in four stages - count words, lines and sentences, find the most frequent and longest words with unordered_map, and print a sorted report.

Word counters look trivial until you write one. Is "notebook," the same word as "notebook"? Is "couldn't" one word or two? Does "knew..." end three sentences or one? How many characters are in "cafรฉ"? Every text tool, from a word processor's status bar to a search engine, has to answer questions like these, and the answers are design decisions, not facts.

In this project you'll build a text analyzer that reads a passage from the Input box and reports its size, its sentences, its most frequent words and its longest words. You'll build it in four stages. Each stage is a complete program containing everything so far, since every run starts fresh, so you can paste your own text into the Input box at any stage and see what it makes of it.

The project uses std::string and std::string_view, character functions from <cctype>, string streams, std::unordered_map and std::unordered_set, sorting algorithms including partial_sort, and formatted output.

The sample text

Every stage uses the same short story as input, written for this project. It's small enough to check by hand and has the awkward cases deliberately built in: punctuation glued to words, apostrophes, an ellipsis, a question and an exclamation, an accented letter, a blank line between paragraphs, and a final sentence with no period.

TEXT
The lighthouse keeper counted the waves every night. She wrote each number in a
small blue notebook, and the notebook never lied!
Why did she count them? Nobody in the village knew... Some said the sea was ...

One winter the waves stopped. The water lay flat and silent for three whole ...
Even the cafรฉ by the harbor fell quiet. On the fourth morning she climbed ...

(Lines are shortened here; the Input box holds the full text, four lines of prose with a blank line in the middle.)

Stage 1: read everything, count the basics

The first decision is how to read. Some statistics are about lines (lines, paragraphs), and others ignore line breaks entirely (a sentence can continue onto the next line). So the program reads the whole input into one string first, and then examines it in different ways. The idiom for that uses stream buffer iterators:

C++
std::string text{std::istreambuf_iterator<char>{std::cin}, std::istreambuf_iterator<char>{}};

The first iterator reads characters from std::cin one at a time, without skipping whitespace; the default-constructed one means "end of stream". The string constructor copies everything between them. For a few kilobytes that's instant; even a whole novel (about a megabyte) fits in memory without trouble.

The second decision is what a character is. text.size() counts bytes. Our text is UTF-8, where English letters take one byte but "รฉ" takes two. To count characters as a reader would, count the bytes that start a character: every UTF-8 continuation byte has the bit pattern 10xxxxxx, so (byte & 0xC0) != 0x80 is true exactly once per character. (Strictly speaking this counts Unicode code points; some emoji and accented letters are built from several code points. For ordinary prose, code points are a good approximation.)

Lines and paragraphs come from reading the text again with std::getline over a std::istringstream. A paragraph starts at any non-blank line that follows a blank line (or the start of the text).

Words, for now, are simply whitespace-separated tokens, read with >>.

C++ Stage 1: bytes, characters, lines and tokens

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with std::cin or std::getline), it gets the next line from this box.

The two counts differ for two reasons: bytes includes the line breaks, which characters leaves out, and the "รฉ" in "cafรฉ" is two bytes but one character. Count the lines: there are five, including the blank one, and two paragraphs.

std::ranges::count_if returns a signed count (the range's difference type), so the code converts it with static_cast to std::size_t before mixing it with text.size(). Mixing signed and unsigned numbers in arithmetic is a classic source of bugs, and making the conversion explicit shows you've thought about it (see Integer Types).

Stage 2: real words and their frequencies

A token isn't a word yet. The tokens include notebook,, lied! and knew..., and The and the are different strings. Stage 2 turns tokens into words with normalize:

  1. Trim characters that can't be part of a word from both ends. Inside a word, keep everything, so couldn't and keeper's stay whole. That's the rule most readers would expect.
  2. Lowercase what's left.

Which characters can be part of a word? Letters and digits, obviously, which std::isalnum tests. But std::isalnum looks at single bytes in the C locale, and both bytes of "รฉ" are above 127, so it says "no". A trimming rule built only on isalnum would cut "cafรฉ" down to "caf". The fix is to treat every byte of 128 or more as part of a word. That's crude, but it's correct for every letter outside ASCII, and it's honest about the trade-off: a non-ASCII punctuation mark such as "โ€”" would also count as part of a word.

Note the casts to unsigned char before calling std::isalnum and std::tolower. These functions require a value that fits in unsigned char (or EOF). On platforms where char is signed, the bytes of "รฉ" are negative, and passing a negative value is undefined behavior. Casting first is one of those habits that separates code that works from code that works on your machine.

Counting with a hash map. Each word goes into a std::unordered_map<std::string, int>. ++counts[word] does all the work: operator[] inserts the word with a count of 0 the first time it's seen, then the ++ increments it. Lookups and insertions take constant time on average, so counting a million words takes about a million cheap steps.

Getting the top N. A hash map has no useful order; iterating it visits words in an unspecified order that depends on the hash function and the bucket count. To print the most frequent words, copy the entries into a vector and sort them. You don't need the whole vector sorted, only the first N, and std::ranges::partial_sort does exactly that: it puts the N smallest elements (by the comparison) at the front, in order, and leaves the rest in an unspecified order. For a large vocabulary that's noticeably cheaper than a full sort.

The comparison sorts by count, descending, and breaks ties alphabetically. Without a tie-breaker, words with equal counts would come out in whatever order the hash map produced, which can differ between compilers or library versions. Deterministic output is worth one extra line.

C++ Stage 2: normalized words and the top ten

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with std::cin or std::getline), it gets the next line from this box.

The top of the list is dominated by little words: "the" appears 15 times, far more than anything else, followed by "and" and "she". That's normal for any English text, and it's the reason real tools filter out stop words, which is part of stage 4.

A few things to notice in the code:

  • using WordCounts = ... gives the long map type a name that says what it holds (see Type Aliases).
  • The vector is built straight from the map's iterators. The map's elements are pair<const std::string, int>, and each converts to the vector's pair<std::string, int>. The key can't stay const in the vector, because sorting has to move elements around.
  • entries.begin() + static_cast<std::ptrdiff_t>(n) makes the size-to-offset conversion explicit, for the same reason as in stage 1.
  • std::min(n, entries.size()) protects against texts with fewer than ten distinct words. partial_sort with a middle iterator past the end would be undefined behavior.

Stage 3: sentences, averages and the longest words

Sentences are the trickiest thing to count. A first idea, "count the ., ! and ? characters", gives the wrong answer here: "knew..." would count as three sentences. And the last sentence has no punctuation at all.

The rule this program uses: a sentence ends at a ., ! or ? that is followed by whitespace or the end of the text, but only if the sentence actually contained a word. Walk through the cases:

  • night. She ends a sentence: the period is followed by a space.
  • knew... Some ends one sentence. The first two dots are followed by more dots, so they don't count. The third is followed by a space, so it does.
  • A number like 3.5 doesn't end a sentence, because the dot is followed by a digit.
  • again at the very end has no punctuation, so after the loop, a sentence that has started but not ended is counted too.

It still gets abbreviations wrong: "Dr. Lee arrived." counts as two sentences. Handling that properly needs a list of abbreviations, or a statistical model, which is why industrial-strength sentence splitters are surprisingly large. Our rule is right for most plain prose, and its limitation is documented. That's a reasonable engineering result.

Averages come from totals collected while counting words: the number of letters in all words (in characters, using utf8_length from stage 1, so "cafรฉ" has 4 letters) and the number of sentences. Note the casts to double before dividing. Integer division would silently drop the fraction.

Longest words are found among the distinct words, the keys of the map. Collecting them as std::string_views avoids copying strings: the views point into the map's keys, which stay put as long as the map isn't modified. They're sorted by length (descending) and then alphabetically, and the first five are printed.

C++ Stage 3: sentences, averages and longest words

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with std::cin or std::getline), it gets the next line from this box.

The story has 10 sentences: the ellipsis counts once, and the unpunctuated last sentence still counts. The three samples at the end show the rule's strengths and its known weakness: Wait... what?! No. gives 3, Pi is 3.14 or so gives 1, and Dr. Lee arrived. gives 2 instead of 1.

The sort comparison calls utf8_length several times per comparison. For a vocabulary of a few hundred words, that's nothing. For a big corpus you'd compute each length once, store (length, word) pairs and sort those, which is the "decorate, sort, undecorate" pattern. As usual, measure before optimizing (see Writing Fast C++).

Stage 4: stop words and the final report

The last stage ties it together into one report. Following the pattern from the earlier projects, the program computes first and prints second. analyze reads the text and fills a TextStats struct; print_report only formats it. Splitting it this way means you could test analyze on strings without any input, or print the same statistics as JSON later.

Stop words. The most-common list now skips very common function words ("the", "and", "she", ...). They're stored in a std::unordered_set<std::string_view>: membership tests take constant time on average, and std::string_view keys are fine because they point at string literals, which live for the whole program. Looking up a std::string in that set works because std::string converts to std::string_view implicitly. Stop words are only filtered from the ranking; all other statistics still count every word. Choosing a stop list is a judgment call, and every text-processing library ships a different one.

New measures.

  • Lexical diversity is distinct words divided by total words. A value near 1 means almost every word is new; repetitive text scores lower. It depends strongly on text length (long texts always repeat more), so only compare texts of similar size.
  • Words used once (linguists call them hapax legomena) are counted with std::ranges::count_if over the map's values.
  • Reading time assumes 230 words per minute, a common middle estimate for adults reading prose silently. It's rounded up, with a minimum of one minute, because "0 minutes" is never a helpful answer.

Bars. The frequency chart scales each bar against the top count, so the longest bar is always 20 characters wide regardless of the text's length.

C++ Stage 4: the complete text report

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with std::cin or std::getline), it gets the next line from this box.

With the stop words gone, the ranking finally says something about the story: notebook, waves and count rise to the top, followed by words that appear once, in alphabetical order. (A longer text gives a more interesting ranking.) Paste any text of your own into the Input box and run it again; a news article, a song you wrote, or the README of a project you're working on.

Why analyze takes std::string_view but builds strings for the streams

analyze takes std::string_view so callers can pass a std::string, a literal or part of a larger buffer. std::istringstream, however, only accepts a std::string (it keeps its own copy), so the function creates one with std::string{text} for each stream. C++23 also added std::ispanstream in <spanstream>, which reads directly from existing characters without copying; it's a nice upgrade for large inputs (see String Streams).

const double words = s.words;

Converting the word count to double once, at the top of print_report, means every later division is floating-point automatically. It avoids sprinkling static_cast<double> everywhere, and avoids the bug where one forgotten cast quietly turns 97 / 10 into integer division (9 instead of 9.7).

Lookup with a string in a set of string_views

stop_words.contains(word) passes a std::string to a set whose keys are std::string_view. That compiles because std::string converts to std::string_view implicitly. The opposite (a set of std::string looked up with a std::string_view) creates a temporary string per lookup, unless you enable heterogeneous lookup with a transparent hash and std::equal_to<> (C++20), as shown in std::unordered_map.

Ideas to extend it

  • Top bigrams. Count pairs of consecutive words ("the waves", "the notebook") in a second map. Reset the pair at sentence ends so pairs don't span sentences.
  • Syllables and readability. Estimate syllables by counting groups of vowels in each word, then compute the Flesch reading-ease score: 206.835 โˆ’ 1.015 ร— (words / sentences) โˆ’ 84.6 ร— (syllables / words).
  • Better sentences. Keep a small set of abbreviations (dr, mr, mrs, e.g, etc) and don't end a sentence after them. Test it with the samples from stage 3.
  • Compare two texts. Split the input at a line containing only ---, analyze both halves and print their shared words, using set intersection on sorted word lists (see std::set and std::unordered_set).
  • Faster counting. Generate a large text (repeat the story 10,000 times in a loop), time analyze with <chrono>, then try counts.reserve(...) and a transparent hash to avoid creating strings in lookups (see Dates, Times and Durations with chrono).
  • Ranges everywhere. Rewrite the token loop as a pipeline: std::views::split on whitespace, transform(normalize), filter out empty words, and count with a range-based for.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. For the string "cafรฉ", what do text.size() and the program's utf8_length(text) return?

    UTF-8 stores รฉ in two bytes, so size() is 5. utf8_length counts only bytes that start a character (not 10xxxxxx continuation bytes), giving 4.
  2. 2. Why does is_word_char cast the char to unsigned char before calling std::isalnum?

    Where char is signed, bytes above 127 (like those of รฉ) are negative. Passing a negative value other than EOF to the cctype functions is undefined behavior, so cast to unsigned char first.
  3. 3. How many sentences does count_sentences find in "She paused... then left. It was 3.5 km"?

    The ellipsis after paused ends sentence one, because its last dot is followed by a space. "left." ends sentence two. In 3.5 the dot is followed by a digit, so it does not count, and the unpunctuated tail is sentence three.
  4. 4. Why does top_words break ties alphabetically?

    Iteration order of a hash map depends on the hash function and bucket count. A tie-breaker makes the output deterministic everywhere.
  5. 5. What does std::ranges::partial_sort(v, v.begin() + 3, comp) guarantee?

    partial_sort sorts only the prefix you ask for, which is cheaper than a full sort when you need just the top N. The guarantee about a single position is what std::nth_element provides.

Summary

  • Read the whole input once with std::istreambuf_iterator, then examine it as lines, tokens or raw characters as each statistic requires.
  • size() counts bytes. Count UTF-8 characters by skipping continuation bytes ((c & 0xC0) == 0x80), and remember that byte-based <cctype> functions only understand ASCII.
  • Always cast to unsigned char before calling std::isalnum, std::tolower and friends.
  • A word is a design decision: trim punctuation from the ends, keep apostrophes inside, lowercase, and document the limits (abbreviations, non-ASCII punctuation).
  • Count with ++counts[word] in a std::unordered_map, then copy to a vector and use std::ranges::partial_sort with a deterministic tie-breaker for the top N.
  • Keep analysis (analyze filling a struct) separate from presentation (print_report), and convert to double once before dividing.

Next: Project: Tic-Tac-Toe, where you'll model a board with std::array and enum class, read moves from the Input box and build an unbeatable computer player with minimax.