Skip to content
Tutorials Center
My learning

Topics

Lesson 190 of 201 ยท Projects

Project: Word Counter

Build a Java word counter step by step. Tokenize text with a regex, count with Collectors.groupingBy, sort a top-N report, read files and stdin, and write a small CLI.

Counting words is the "hello world" of data processing, and it is a perfect project for learning streams. The idea fits in one sentence (split text into words, count each distinct word, show the most frequent), yet the details are where the real skills live: what counts as a word, how to ignore case, how to rank ties fairly, and how to get text in from a file or from standard input.

You'll build it in four stages: tokenizing, counting with groupingBy, ranking the top N, and then reading real input. It uses streams, collectors, regular expressions and the Files API. If you've built the Text Analyzer, this one goes deeper on the streams side.

Stage 1: what is a word?

Splitting on spaces is the first idea and the first mistake. "Hello, world!" would give Hello, and world!, with the punctuation glued on, and Hello and hello would count as different words. A better definition: a word is a run of letters, digits or apostrophes, compared in lower case.

Java Stage 1: tokenizing with a regex

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Matcher.results() returns a stream of MatchResults, one per match, which feeds straight into the rest of a pipeline. \p{L} matches any letter in any language, which is why Cafรฉ stays one word. We lower-case with Locale.ROOT so the result doesn't depend on the user's language settings (in Turkish, "I".toLowerCase() would give a dotless ฤฑ). The apostrophe lets it's and isn't survive as single words, and the replaceAll removes quotes that merely wrap a word, as in 'hello'.

Stage 2: counting with groupingBy

Counting is the textbook use of Collectors.groupingBy with Collectors.counting(): group equal words together, and for each group report how many there are.

Java Stage 2: frequency table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

groupingBy(w -> w, counting()) reads like the sentence it implements: key each word by itself, then count each group. The result is a Map<String, Long>. Notice the loop version in the last lines; Map.merge(w, 1, Integer::sum) does the same job and is just as idiomatic. The stream version shines when you want to chain more steps, and the loop version when you need a HashMap<String, Integer> or want to avoid boxing to Long.

A plain groupingBy gives a HashMap, so the iteration order is arbitrary. Any report that people read needs an explicit sort.

Stage 3: the top N, with fair tie-breaking

Sorting by count descending is easy. What's easy to forget is ties: "fox" and "the" may both appear four times, and an unstable order makes your output change between runs and break tests. Sort by count descending, then alphabetically, and the order is fully determined.

Java Stage 3: ranked report

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The comparator is the interesting line. Map.Entry.<String, Long>comparingByValue().reversed() sorts by count descending; the explicit type witness <String, Long> is needed because the compiler can't infer it through .reversed(). .thenComparing(Map.Entry.comparingByKey()) breaks ties alphabetically. See Comparable and Comparator for the general pattern.

A limit(n) after sorted stops the stream early only conceptually: sorted still has to see every element. For a huge number of distinct words you could keep a bounded PriorityQueue of size N instead, which uses O(N) memory, but for ordinary documents the sorted stream is simple and fast enough.

The stop-word filter removes words like "the" and "and" that dominate every text and tell you nothing. It's a Set so each lookup is constant time, and a boolean decides whether to apply it.

Stage 4: real input from a file or standard input

A command-line tool should read from a file if given one, and otherwise from standard input, so it works in pipelines (cat book.txt | wordcount). Files.lines streams a file line by line, so even a very large file never has to fit in memory.

The playground has no files of its own, so this program first writes a small sample file to a temporary location, then counts it through a streaming pipeline. The standard-input path reads from the Program input box.

Java Stage 4: the complete word counter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Three points to notice:

  • try (lines) closes the stream. Files.lines holds an open file handle, so it must be closed, and a try-with-resources on an existing variable is the shortest way.
  • flatMap turns each line into a stream of matches, so the pipeline works line by line and the memory use stays flat however big the file is.
  • in.lines() is also a Stream<String>, so the same count method serves both sources. Designing around Stream<String> rather than around files is what makes the code reusable and easy to test.

A word count only needs to hold the distinct words in memory, not the whole text. For English that's a few tens of thousands of entries even for a long book, which is why this approach scales so comfortably.

A small command-line shell

Turn it into a real tool by reading arguments in main(String[] args). Typical options are a file name, --top N and --ignore-stop-words. A tidy structure is: parse the arguments into a small record (Options(Path file, int top, boolean skipStop)), call count, then call print. For parsing flags see command-line arguments.

Ideas to extend it

  1. Count bigrams (pairs of consecutive words) with IntStream.range over the word list.
  2. Report the longest word, the average word length and the number of lines.
  3. Use Collectors.groupingBy(String::length, TreeMap::new, Collectors.counting()) to build a word-length histogram.
  4. Process several files in parallel with parallel(), merging maps with Collectors.toConcurrentMap or groupingByConcurrent.
  5. Write the report to a CSV file with Files.write.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is text.split(" ") a poor way to find words?

    Splitting on a space leaves tokens like "world!" and keeps "Hello" and "hello" apart. A regex of letters and digits plus lower-casing gives a clean word list.
  2. 2. What does Collectors.groupingBy(w -> w, Collectors.counting()) return for a stream of strings?

    groupingBy builds a Map keyed by the classifier result, and counting() is the downstream collector that reduces each group to its size.
  3. 3. Why sort ties by key after sorting by count?

    A deterministic total order needs a second criterion. thenComparing(comparingByKey()) makes the report repeatable and testable.
  4. 4. What must you do with a stream returned by Files.lines(path)?

    Files.lines keeps the file open until the stream is closed. A try-with-resources block guarantees the handle is released even if an exception occurs.
  5. 5. Why lower-case with toLowerCase(Locale.ROOT)?

    The no-argument toLowerCase uses the default locale. Locale.ROOT gives the same result everywhere, which is what you want for comparing words.

Summary

  • Define a word with a regex ([\p{L}\p{N}']+) and normalize with toLowerCase(Locale.ROOT); Matcher.results() feeds a stream directly.
  • groupingBy(w -> w, counting()) builds the frequency map; Map.merge(w, 1, Integer::sum) is the loop equivalent.
  • Rank with comparingByValue().reversed().thenComparing(comparingByKey()) so ties come out in a stable, readable order.
  • Design around Stream<String> so the same counting code serves files (Files.lines) and standard input (BufferedReader.lines), and close file streams with try-with-resources.
  • Memory depends on the number of distinct words, not on the file size, because lines are processed one at a time.

Next: Data Structures in Java, where you will build the lists, stacks, queues and trees behind many of the techniques used in these projects.