Skip to content
Tutorials Center
My learning

Topics

Lesson 177 of 201 ยท Projects

Project: Text Analyzer

Build a Java text analyzer in stages. Count words and sentences with regular expressions, rank word frequencies with HashMap and streams, and print a full report.

Word counters, readability checkers, search engines, spam filters and plagiarism detectors all start the same way: they take raw text and turn it into numbers. How many words? Which ones appear most often? How long are the sentences? Answering those questions well is harder than it looks, because human text is full of punctuation, apostrophes, capital letters and odd spacing, and every one of them can quietly throw a count off.

In this project you'll build a text analyzer in five stages. It reads any text you paste into the Input box and reports basic counts, word frequencies (with and without common filler words), sentence statistics, the longest words and a small chart. Every stage is a complete program; paste in an email, a news article or a page from a book and see what it says.

You'll practice string methods, regular expressions, HashMap and HashSet, sorting with comparators, streams and collectors. This project uses compact source files (void main()), since it's mostly a set of functions over text; the earlier projects used classes because they modeled things with state.

Stage 1: reading the text and counting the basics

Until now, programs have read input line by line. A text analyzer wants all of it at once, as one string. The shortest way is to read every byte from standard input and decode them as UTF-8:

Java Stage 1: reading all input and basic counts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

System.in.readAllBytes() returns everything on standard input as a byte[], and new String(bytes, StandardCharsets.UTF_8) turns those bytes into text. Naming the charset explicitly is a good habit: bytes only become characters through an encoding, as the Character Sets and Encoding lesson explains. readAllBytes can throw a checked IOException, so main declares throws IOException. (Compact source files import all of java.base automatically, which is why IOException, StandardCharsets and List need no import lines.)

The basic counts each use one string tool:

  • Lines: text.lines() splits on any line terminator (\n, \r\n or \r), so it works with text from any operating system.
  • Paragraphs: paragraphs are separated by a blank line. The regex \\n\\s*\\n matches a line break, optional whitespace and another line break, so a "blank" line that contains a stray space still counts. strip() first removes leading and trailing whitespace, which would otherwise create empty paragraphs at the edges.
  • Characters without spaces: replaceAll("\\s", "") deletes every whitespace character (spaces, tabs, line breaks) and length() counts what's left.

The rough word count splits on whitespace, and the sample at the bottom shows why it's only rough: years. and night, include their punctuation. Counted this way, lamp and lamp. would be two different words. Stage 2 fixes that.

length() counts UTF-16 code units, not visible characters. For ordinary text they're the same, but an emoji like ๐ŸŽ‰ counts as 2. If you need exact character counts for any text, use text.codePointCount(0, text.length()); see Characters and Booleans.

Stage 2: finding real words with a regex

Instead of splitting text on what words are not (whitespace), it's more reliable to describe what a word is and find every match. That's a job for Pattern and Matcher.

What is a word? Letters, certainly, and digits too (2026, 4K). Apostrophes inside a word belong to it (doesn't, keeper's), and so do hyphens between letters (well-known). Here's that definition as a regex:

TEXT
[\p{L}\p{N}]+(?:['โ€™-][\p{L}\p{N}]+)*

Read it as: one or more letters or digits, followed by any number of groups made of an apostrophe (straight or curly) or hyphen and more letters or digits. \p{L} means "any letter in any language", so cafรฉ and naรฏve are single words, which [a-zA-Z] would get wrong.

Java Stage 2: words as regex matches

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The tricky sample shows the pattern's decisions at work: don't and it's stay whole, well-known is one word, cafรฉ keeps its accent, 24/7 becomes two numbers, the lone -- and the ... disappear, and the quotes around 'Really?' are dropped because an apostrophe only counts between letters. Every decision here is a choice, not a fact; a search engine might split well-known into two words, and that's fine as long as you choose on purpose.

For the lighthouse text, the real count is the same as the rough count, because this text has no stray punctuation tokens. What changed is the words themselves: years. is now years and The is now the, so later counting will treat them as the same word as their other occurrences.

Implementation notes:

  • The pattern is compiled once, into a static final field. Pattern.compile does real work (it turns the regex into an internal matcher program), so compiling inside a loop or a frequently called method wastes time. String.split and String.matches compile their regex on every call, which is fine for a one-off but not in hot code.
  • matcher.find() in a while loop moves through the text match by match, and group() returns the current match. This is the standard way to extract all matches; see Regular Expressions.
  • Words are lowercased with Locale.ROOT. Counting needs The and the to be the same word. As in the library project, Locale.ROOT guarantees the same result on every machine.
  • new HashSet<>(words).size() counts distinct words in one line, because a set keeps only one copy of each element.

Stage 3: word frequencies

Now the classic part: how often does each word appear? A HashMap<String, Integer> from word to count is the natural structure, and merge makes counting a one-liner. Sorting the results is the interesting design question.

Java Stage 3: counting and ranking words

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The first list is dominated by the, and and at, words that appear in every English text and say nothing about this one. Removing them reveals what the text is about: the keeper, the lamp, the stairs, the lighthouse. Search engines and keyword tools have relied on this kind of filtering for decades; the filler words are called stop words.

The techniques:

  • counts.merge(word, 1, Integer::sum) means: if word isn't in the map, put 1; otherwise replace the old count with old + 1. It replaces the four-line "get, check for null, add, put" pattern. The HashMap lesson compares it with getOrDefault and compute.
  • Ties are broken alphabetically. Sorting by count alone would list words with equal counts in hash order, which looks random and can change between Java versions. thenComparing(Map.Entry.comparingByKey()) makes the order fully deterministic, which matters for anything you test or compare.
  • The explicit <String, Integer> type arguments on comparingByValue() are needed because .reversed() follows it, the same inference limit seen in the library project.
  • Removing keys through keySet(). A map's keySet() is a live view: removing elements from it removes the matching entries from the map. meaningful.keySet().removeAll(STOP_WORDS) deletes every stop word in one call. We copy counts first, so the original totals stay intact.
  • Set.of(...) for constants. It creates an immutable set with fast contains. One trap: Set.of throws IllegalArgumentException if you list the same element twice, so a typo like writing "the" twice in a long list crashes the program at startup. That's actually helpful, since the duplicate was a mistake anyway.

Notice that keeper and keeper's are counted separately. Treating them as one word requires stemming (reducing words to a root form), a deep topic in language processing. It's in the ideas list at the end.

Stage 4: sentences and word lengths

Word counts describe vocabulary; sentence statistics describe style. Short sentences read fast, long ones slow. This stage splits the text into sentences and computes sentence lengths, the longest words, the average word length, vocabulary richness and an estimated reading time.

Java Stage 4: sentences, longest words and averages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Splitting sentences with a lookbehind

The regex (?<=[.!?])\\s+ splits on whitespace, but only whitespace that comes right after a period, exclamation mark or question mark. The (?<=...) part is a lookbehind: it checks what precedes the match without including it in the match, so the punctuation stays attached to its sentence instead of being thrown away by the split.

Requiring whitespace after the punctuation also keeps numbers like 3.5 and web addresses intact. It's still a heuristic: Dr. Lee arrived. would be split after Dr., and a sentence ending in a quotation mark needs extra care. Professional tools use java.text.BreakIterator or language models for this; the regex handles ordinary prose well, and knowing its limits is part of using it.

Statistics with streams

  • IntSummaryStatistics collects count, sum, minimum, maximum and average in a single pass. mapToInt(Integer::intValue) converts the List<Integer> into an IntStream first, because only primitive streams have summaryStatistics().
  • Finding the longest sentence reuses the lengths list: lengths.indexOf(stats.getMax()) gives the position of the first sentence with the maximum word count, and the same index picks it from sentences. The two lists are parallel, just like the arrays in the grade book.
  • The longest words come from distinct() (so lighthouse isn't listed twice), then a comparator on length, descending, with alphabetical order breaking ties. Comparator.comparingInt(String::length) avoids boxing each length.
  • average() returns an OptionalDouble, empty when there are no words, so orElse(0) makes empty input safe.
  • Percentages need floating-point math. 100.0 * distinct / words.size() starts with a double, so the whole expression is computed in double. 100 * distinct / words.size() would use integer division and truncate.

Stage 5: the full report

The last stage combines everything into one report and adds two visual touches: a bar chart of the most frequent meaningful words and a distribution of word lengths. The analysis is collected into a TextStats record first, and the report is printed from it, so calculating and presenting stay separate.

Java Stage 5: the complete text analyzer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Replace the lighthouse story with your own text and run it again. Try a paragraph of very short sentences, then one long, winding sentence, and watch the words-per-sentence line change.

Analysis and presentation, separated

analyze computes, printReport prints. Between them sits TextStats, a record that holds the results. This split pays off in several ways: you could print the same stats as JSON or HTML without touching the analysis, write a test that checks analyze returns the right numbers without capturing printed output, or compare two texts by analyzing both. Mixing calculations into print statements makes all of that hard.

WordCount replaces Map.Entry<String, Integer> from Stage 3. Entries work, but wc.word() and wc.count() read better than getKey() and getValue(), and the comparator chain comparingInt(WordCount::count).reversed().thenComparing(WordCount::word) needs no explicit type arguments because method references carry their own types.

New statistics

  • Word length distribution: groupingBy(String::length, TreeMap::new, counting()) maps each length to how many words have it, with TreeMap::new keeping the lengths in ascending order for the chart. In English prose the peak is usually at three or four letters.
  • Words used once: the values of the count map that equal 1. Linguists call such words hapax legomena; a high share of them suggests a rich vocabulary (or a short text).
  • Longest words now come from counts.keySet(), which is already distinct, so distinct() is no longer needed.

Bars that scale

The top-word chart scales every bar relative to the most frequent word: 20.0 * count / maxCount gives the top word 20 blocks and the others proportionally fewer, so the chart fits on screen whether the top count is 6 or 6,000. getFirst() (a sequenced collection method from Java 21) reads the first, largest entry. The word-length chart scales the same way, using Collections.max to find its largest count; without scaling, a long text would print bars hundreds of characters wide.

The empty case

If the input has no words, analyze would still run, but getFirst() on an empty list throws NoSuchElementException, and 100.0 * distinct / words.size() would divide by zero and print NaN. Rather than sprinkling checks through the report, main checks once and explains what to do. Try it: clear the Input box and press Run.

Ideas to extend it

  • Word search with context. Add a search term and print every sentence containing it, with the word highlighted in brackets. A regex with \\b word boundaries and Pattern.quote for the search term keeps it safe.
  • Simple stemming. Strip common endings ('s, s, ed, ing) before counting, so keeper's counts toward keeper and climbed toward climb. You'll quickly discover why real stemmers have many rules (stairs should become stair, but lens shouldn't become len).
  • Bigrams. Count pairs of consecutive words (the keeper, the lamp) with the same merge idea, using words.get(i) + " " + words.get(i + 1) as the key.
  • Readability score. Estimate syllables by counting vowel groups per word, then compute a Flesch reading-ease score from words per sentence and syllables per word.
  • Compare two texts. Split the input on a line containing only ---, analyze both halves and print which words are unique to each, using set difference from HashSet.
  • Character frequency. Count letters with an int[26] or a TreeMap<Character, Integer> and show which letters are most common.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does WORD.matcher("It's well-known -- 24/7!") find, using the pattern from Stage 2 (before lowercasing)?

    An apostrophe or hyphen between letters keeps a word together, so It's and well-known stay whole. A slash isn't part of the pattern, so 24/7 becomes two matches, and -- has no letters or digits, so it isn't a word.
  2. 2. Why does the program compile its regex into a static final Pattern field?

    Pattern.compile parses and prepares the regex. Doing it once and reusing the Pattern avoids repeating that work, which String.split and String.matches redo each time they're called.
  3. 3. After counts.merge("lamp", 1, Integer::sum) runs three times on an empty map, what is counts.get("lamp")?

    The first merge inserts 1 because the key is absent. Each later merge replaces the old value with Integer.sum(old, 1), giving 2 and then 3.
  4. 4. What does "Hi. It costs 3.50 now! Ok?".split("(?<=[.!?])\\s+") produce?

    The split happens only at whitespace that follows . ! or ?. The period inside 3.50 isn't followed by whitespace, so the number stays intact, and the lookbehind keeps the punctuation attached to each sentence.
  5. 5. Why does the Stage 3 ranking add thenComparing(Map.Entry.comparingByKey())?

    Many words share the same count. Breaking ties by the word itself makes the output deterministic, which matters for reading, testing and comparing results.

Summary

  • new String(System.in.readAllBytes(), StandardCharsets.UTF_8) reads all input as one string; lines(), strip(), split and replaceAll give quick basic counts.
  • Define words by what they are: a precompiled Pattern with \p{L} and \p{N}, found in a matcher.find() loop and lowercased with Locale.ROOT.
  • HashMap.merge(word, 1, Integer::sum) counts frequencies; sort entries by count descending with an alphabetical tie-breaker for deterministic output.
  • Stop words in an immutable Set.of(...) remove filler words, and keySet().removeAll(...) deletes them from a map in one call.
  • A lookbehind regex (?<=[.!?])\\s+ splits sentences while keeping their punctuation; IntSummaryStatistics, groupingBy with TreeMap::new and distinct() produce the statistics.
  • Keep analysis and presentation separate with a result record, and handle the empty input once, up front.

Next: Project: Tic-Tac-Toe, where you'll build a complete game with a 2D board, enums and a computer opponent.