Skip to content
Tutorials Center
My learning

Topics

Lesson 43 of 48 ยท Projects

Project: Word Frequency Counter

Build a word frequency counter in Elixir: stream a file line by line, split words with a regex, count with Enum.frequencies and sort the top results.

Counting words is a small problem that touches many everyday Elixir tools: lazy streams for reading, regular expressions for splitting, Enum.frequencies/1 for counting and Enum.sort_by/3 for ranking. In this project you will build a program that reads a text, finds the most common words and prints a tidy report.

The real Elixir code below is read-only, because Elixir is not installed on this site. Each step also has an editable JavaScript stand-in that follows the same design, so you can experiment with the idea and see real output. Where a stand-in differs from Elixir, the text says so. If you need a refresher first, see The Enum Module, Lazy Processing with Stream and Sigils and Regular Expressions.

The plan

Read Stream lines from a file.
Split A regex finds the words.
Count Enum.frequencies does the work.
Rank Sort by count, then by word.

Four small steps, each one a pure function.

1 / 2

Design: small pure functions

A good Elixir program separates work that talks to the outside world (files, the screen) from work that only transforms data. The transformations are plain functions that take data and return data, so they are easy to test and to combine with the pipe operator (see The Pipe Operator).

We will build one module with these functions:

  • words/1 turns a string into a list of lowercase words.
  • count/1 turns a list of words into a map of word to count.
  • top/2 returns the n most common words as a sorted list of tuples.
  • report/1 formats the result as text.

Step 1: extracting words

ELIXIR
defmodule WordCounter do
  @stop_words ~w(the a an and of to in is it)

  def words(text) do
    ~r/[\p{L}']+/u
    |> Regex.scan(String.downcase(text))
    |> List.flatten()
  end

  def without_stop_words(words) do
    Enum.reject(words, &(&1 in @stop_words))
  end
end

~r/[\p{L}']+/u matches runs of Unicode letters and apostrophes. The u modifier turns on Unicode mode, so \p{L} also matches letters such as รฉ or รผ, not only a to z. Regex.scan/2 returns a list of lists (one inner list per match, with its capture groups), so List.flatten/1 produces a plain list of strings. @stop_words is a module attribute holding a list made with the ~w sigil, and &(&1 in @stop_words) is a capture that asks whether a word is in that list.

Here is the same idea in JavaScript. A regex with the u flag and match plays the role of Regex.scan.

JS Step 1: split text into words (JavaScript stand-in)

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice that match returns null when there is nothing to find, so the stand-in adds ?? []. In Elixir Regex.scan/2 simply returns [].

Step 2: counting

Counting is one call in Elixir:

ELIXIR
def count(words), do: Enum.frequencies(words)
ELIXIR
iex> Enum.frequencies(["fox", "dog", "fox"])
%{"dog" => 1, "fox" => 2}

Enum.frequencies/1 walks the list once and builds a map from each element to the number of times it appears. You could write the same thing yourself with Enum.reduce/3 and Map.update/4, which is a good exercise:

ELIXIR
def count(words) do
  Enum.reduce(words, %{}, fn word, acc ->
    Map.update(acc, word, 1, &(&1 + 1))
  end)
end

Map.update/4 takes the map, the key, a default used when the key is missing, and a function applied when the key exists. Because maps are immutable, each step returns a new map and the old one is untouched (see Immutability and Rebinding).

Step 3: ranking

A map has no meaningful order, so to get a ranking we convert it to a list of {word, count} tuples and sort:

ELIXIR
def top(counts, n) do
  counts
  |> Enum.sort_by(fn {word, count} -> {-count, word} end)
  |> Enum.take(n)
end

The sort key is a tuple {-count, word}. Elixir compares tuples element by element, so higher counts come first (the minus sign flips the order) and words with the same count are sorted alphabetically. Without the tie-breaker, the order of equal counts would depend on the internal order of the map, which you should never rely on.

Sorting by a tuple key is the standard way to sort by several criteria in Elixir. See Ranges, Sorting and Custom Ordering for more.

Now let us build the whole pipeline in JavaScript and print a small bar chart.

JS Steps 2 and 3: count, rank and report (JavaScript stand-in)

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The output is sorted by count, and dog and fox (two each) appear in alphabetical order. A JavaScript Map keeps insertion order, but we still sort explicitly, exactly as we do in Elixir.

Step 4: reading a big file lazily

If the file is large, File.read!/1 would load all of it into memory. A stream reads one line at a time instead:

ELIXIR
def count_file(path) do
  path
  |> File.stream!()
  |> Stream.flat_map(&words/1)
  |> Stream.reject(&(&1 in @stop_words))
  |> Enum.frequencies()
end

File.stream!/1 returns a lazy stream of lines. Stream.flat_map/2 turns each line into its words, Stream.reject/2 drops stop words, and nothing is read until Enum.frequencies/1 pulls the data through. Memory use depends on the number of distinct words, not on the file size, because lines are processed and forgotten one by one. This stand-in uses a generator function, JavaScript's lazy sequence:

JS Step 4: process lines lazily (JavaScript stand-in)

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The "reading" messages appear only when the loop pulls values, not when the pipeline is created. That is the same laziness as Stream.

Running it

A small entry point turns the counts into a printed report. In a script (word_counter.exs) you could read the path from the command line:

ELIXIR
[path | _] = System.argv()

path
|> WordCounter.count_file()
|> WordCounter.top(10)
|> Enum.each(fn {word, count} ->
  IO.puts("#{String.pad_trailing(word, 12)} #{count}")
end)

Run it with elixir word_counter.exs notes.txt. System.argv/0 returns the arguments as a list of strings, and String.pad_trailing/2 aligns the columns.

Ideas to extend it

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the sort key in top/2 use the tuple {-count, word}?

    Tuples compare element by element: negating the count puts higher counts first, and the word breaks ties alphabetically.
  2. 2. What does Enum.frequencies(["a", "b", "a"]) return?

    It returns a map from each element to how many times it appears.
  3. 3. Why use File.stream!/1 instead of File.read!/1 for a very large file?

    A stream yields one line at a time, so only the current line and the counts need to be in memory.
  4. 4. What does the u modifier in ~r/[\p{L}']+/u do?

    Unicode mode lets the regular expression understand Unicode properties and UTF-8 text.

Summary

  • Keep transformations as small pure functions and combine them with the pipe operator.
  • Regex.scan/2 extracts words, Enum.frequencies/1 counts them and Enum.sort_by/2 with a tuple key ranks them with a stable tie-breaker.
  • File.stream!/1 plus Stream functions process big files without loading them.
  • The JavaScript editors mirror the design; the Elixir blocks are real code you can run locally with elixir.

Next up: Project: Chat Room with Processes.