Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 37 ยท Projects and Practice

Project: Word Frequency Counter in Scala

Build a word frequency counter in Scala 3 step by step: tokenizing, groupMapReduce, ranking, stop words, formatted output, file input and tests, with a runnable Java and Kotlin version.

In this project you will build a small command-line tool that reads a text and prints its most frequent words. It is a classic exercise because it touches the Scala features you have learned: immutable collections, higher-order functions, tuples, string processing, pure functions with side effects at the edge, and tests.

Scala is not runnable on this site, so the Scala code is read-only; copy it into a Scala CLI file or an sbt project (see sbt and Project Setup) to run it. Two runnable stand-ins (Kotlin and Java, not Scala) implement the same design so you can watch it work here.

The plan

Four small steps

Tokenize Split text into lowercase words.
Filter Drop empty strings and stop words.
Count Build a Map from word to count.
Rank Sort by count, then alphabetically.
1 / 2
flowchart LR
    A["Raw text"] --> B["tokenize"]
    B --> C["removeStopWords"]
    C --> D["count"]
    D --> E["top(n)"]
    E --> F["format"]
    F --> G["print"]

Step 1: tokenize

A word is a run of letters (and apostrophes, so don't stays whole). Lowercase everything so The and the count together:

SCALA
def tokenize(text: String): List[String] =
  text.toLowerCase
    .split("[^a-z']+")
    .toList
    .filter(_.nonEmpty)

// tokenize("The cat, the HAT!")  ==  List("the", "cat", "the", "hat")

The regular expression [^a-z']+ means "one or more characters that are not letters or apostrophes", so it splits on spaces and punctuation. A text that starts with punctuation produces an empty first string, which filter(_.nonEmpty) removes.

Step 2: count

The standard library has an operation made for this: groupMapReduce(key)(value)(combine). It groups elements by key, maps each to a value, and reduces each group with combine:

SCALA
def count(words: List[String]): Map[String, Int] =
  words.groupMapReduce(identity)(_ => 1)(_ + _)

// count(List("the", "cat", "the"))  ==  Map("the" -> 2, "cat" -> 1)

identity groups by the word itself, _ => 1 maps every occurrence to 1 and _ + _ adds the ones together. An equivalent version with foldLeft is possible (see Higher-Order Functions), but groupMapReduce states the intention directly.

Step 3: rank

A Map has no order, so convert it to a list of pairs and sort. Sort by count descending and break ties alphabetically, so the output is deterministic:

SCALA
def top(counts: Map[String, Int], n: Int): List[(String, Int)] =
  counts.toList
    .sortBy((word, c) => (-c, word))
    .take(n)

Sorting by the tuple (-c, word) uses the built-in tuple ordering: first by the negated count, then by the word. With the input the cat and the hat and the bat, the top three are (the,3), (and,2) and (bat,1): "bat", "cat" and "hat" all appear once, and "bat" comes first alphabetically.

Step 4: stop words

Common words such as "the" and "and" usually drown out interesting ones. Keep them in a Set for fast membership tests, and make filtering optional by passing the set in:

SCALA
val defaultStopWords: Set[String] =
  Set("the", "a", "an", "and", "or", "of", "to", "in", "is", "it")

def removeStopWords(words: List[String], stop: Set[String]): List[String] =
  words.filterNot(stop.contains)

Passing the set as a parameter keeps the function pure: the caller decides what counts as noise.

Step 5: format the report

SCALA
def format(ranking: List[(String, Int)]): String =
  val width = ranking.map((w, _) => w.length).maxOption.getOrElse(0)
  def bar(n: Int): String = "#" * n
  ranking
    .map((word, c) => s"${word.padTo(width, ' ')}  ${bar(c)} ($c)")
    .mkString("\n")

This builds a bar for each word, padding the words to the same width so the bars line up. The maxOption protects against an empty list (max would throw).

Step 6: the edge

Only main touches the outside world:

SCALA
import scala.io.Source
import scala.util.Using

@main def wordFreq(path: String, n: Int = 10): Unit =
  val text = Using.resource(Source.fromFile(path))(_.mkString)
  val words = removeStopWords(tokenize(text), defaultStopWords)
  println(format(top(count(words), n)))

Using.resource closes the file automatically even if an exception occurs. Because @main functions turn their parameters into command-line arguments, you run it as scala-cli run WordFreq.scala -- book.txt 5. A friendlier version would return Either or Try for a missing file (Option, Either and Try) and print a clear message instead of a stack trace.

Step 7: tests

Pure functions are easy to test with munit (Testing in Scala):

SCALA
class WordFreqSuite extends munit.FunSuite:

  test("tokenize lowercases and drops punctuation") {
    assertEquals(tokenize("The cat, the HAT!"), List("the", "cat", "the", "hat"))
  }

  test("count adds up occurrences") {
    assertEquals(count(List("a", "b", "a")), Map("a" -> 2, "b" -> 1))
  }

  test("ties are broken alphabetically") {
    val counts = Map("pear" -> 2, "apple" -> 2, "fig" -> 5)
    assertEquals(top(counts, 3), List(("fig", 5), ("apple", 2), ("pear", 2)))
  }

  test("empty text gives an empty ranking") {
    assertEquals(top(count(tokenize("")), 5), Nil)
  }

Running the design here

The Kotlin and Java programs below are not Scala, but each step corresponds to the Scala function of the same name.

Kotlin Kotlin: tokenize, count, rank (Kotlin, not Scala)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with readln()), it gets the next line from this box.

Java Java: the same pipeline with streams (Java, not Scala)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Extensions to try

  1. Read from standard input when no file is given (scala.io.StdIn.readLine() in a loop, or Source.stdin).
  2. Count bigrams (pairs of adjacent words) using words.sliding(2).map(_.mkString(" ")).
  3. Add a --min-length option to ignore short words.
  4. Handle accents by normalizing text with java.text.Normalizer before tokenizing.
  5. Process a huge file lazily with Source.getLines() and foldLeft over lines so you never hold the whole text in memory.
  6. Parallelize counting on large inputs (see Threads and Synchronization) by counting per chunk and merging maps with _ ++ _ plus summed values.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does words.groupMapReduce(identity)(_ => 1)(_ + _) produce?

    It groups by the word, maps each occurrence to 1 and sums the ones per group.
  2. 2. Why sort by (-c, word) instead of just -c?

    Maps have no defined order, so equal counts would otherwise come out in arbitrary order.
  3. 3. Why does only main read files and print?

    Pure functions are tested with plain values and no files. See Immutability and Pure Functions.
  4. 4. What is the benefit of maxOption over max here?

    max throws UnsupportedOperationException on an empty collection; maxOption returns None.

Summary

  • The project is a pipeline of small pure functions: tokenize, removeStopWords, count, top, format.
  • groupMapReduce counts occurrences concisely; sort by a tuple for deterministic ranking.
  • Side effects (file reading, printing) live only in main; Using.resource closes files safely.
  • Tests with munit check each function with plain values, including edge cases such as empty input and ties.

Next: Project: Expression Evaluator