Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 36 ยท Projects and Practice

Project: Word Frequency Analyzer

Build a word frequency analyzer in Clojure: read text, tokenize with regex, count with frequencies and reduce, filter stop words, rank, and run it as a command-line tool.

Your first real program should be small, data-oriented and satisfying. A word frequency analyzer fits: it reads text, splits it into words, counts them and prints the most common ones. Along the way you use sequences, maps, reduce, ->> pipelines, regular expressions, sort-by, and file input, and you shape the code as small pure functions that are easy to test.

The Clojure code here is read-only (Clojure does not run in the site's editor); each stage has a JavaScript stand-in you can run to check the logic.

The design

Think of the program as a pipeline of data transformations:

flowchart LR
  A["raw text"] --> B["lowercase"]
  B --> C["tokenize into words"]
  C --> D["remove stop words"]
  D --> E["count each word"]
  E --> F["sort by count, then word"]
  F --> G["take top N and print"]

Each box is a pure function. Only the first (reading) and last (printing) boxes touch the outside world.

Keeping effects at the edges and logic in the middle means you can test every stage with a plain string.

Step 1: tokenize

A word is a run of letters, possibly with an inner apostrophe (don't). re-seq returns all matches of a regex as a sequence:

CLOJURE
(ns wordfreq.core
  (:require [clojure.string :as str]))

(defn tokenize [text]
  (re-seq #"[a-z]+(?:'[a-z]+)?" (str/lower-case text)))

(tokenize "The cat, the HAT. Don't stop!")
;=> ("the" "cat" "the" "hat" "don't" "stop")

Step 2: filter stop words

Very common words ("the", "and") would dominate the result, so we drop them. A set works as a predicate function: (stop-words "the") returns "the" (truthy) or nil.

CLOJURE
(def stop-words
  #{"the" "a" "an" "and" "or" "of" "to" "in" "is" "it" "that" "for" "on" "with"})

(defn significant-words [text]
  (->> (tokenize text)
       (remove stop-words)))

Step 3: count

The standard library already has frequencies, which returns a map from item to count:

CLOJURE
(frequencies ["a" "b" "a"])   ;=> {"a" 2, "b" 1}

To understand it, here is the same thing with reduce: start from an empty map, and for each word increment its count, treating a missing key as 0.

CLOJURE
(defn count-words [words]
  (reduce (fn [counts w] (update counts w (fnil inc 0)))
          {}
          words))

fnil wraps inc so a nil (missing) becomes 0 first. In real code, prefer frequencies.

Step 4: rank

A map has no order, so convert it to a sequence of [word count] pairs and sort. Sorting by [(- count) word] means highest count first and, for ties, alphabetical:

CLOJURE
(defn top-words [freqs n]
  (->> freqs
       (sort-by (fn [[word cnt]] [(- cnt) word]))
       (take n)))

(top-words {"cat" 3 "dog" 3 "emu" 1} 2)
;=> (["cat" 3] ["dog" 3])

Because vectors compare element by element, the sort key gives a deterministic order, which makes output testable.

Step 5: combine and print

CLOJURE
(defn analyze [text n]
  (-> text
      significant-words
      frequencies
      (top-words n)))

(defn format-row [[word cnt]]
  (format "%-12s %3d %s" word cnt (apply str (repeat cnt "#"))))

(defn -main [& [path n]]
  (let [text (slurp (or path "input.txt"))
        n    (Long/parseLong (or n "10"))]
    (doseq [row (analyze text n)]
      (println (format-row row)))))

slurp reads a whole file into a string; doseq runs the printing for each row (never use lazy map for printing). Run it with clj -M -m wordfreq.core story.txt 5 (see tooling).

Step 6: test it

Pure functions give you cheap tests (see testing):

CLOJURE
(deftest analyze-test
  (is (= [["cat" 2] ["hat" 1]]
         (analyze "The cat, the CAT and a hat." 5))))

Extensions to try

  1. Stop words from a file (resources/stopwords.txt), loaded once with delay.
  2. Bigrams (pairs of adjacent words): (partition 2 1 words) then frequencies.
  3. Longest words and average word length with reduce.
  4. Large files: process with line-seq inside with-open and reduce, so memory stays flat (see lazy sequences); realize the work inside the with-open scope.
  5. Parallel counting: split lines into chunks, count each with pmap, then (apply merge-with + partial-counts).
  6. A transducer version: (transduce (comp (map str/lower-case) (mapcat tokenize) (remove stop-words)) (completing count-step) {} lines) (see transducers).

Try it: the whole pipeline in JavaScript

JS Word frequency pipeline

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Try it: extensions

Bigrams, and merging partial counts like (apply merge-with + ...) for parallel counting.

JS Bigrams and merge-with

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does (remove stop-words) work when stop-words is a set?

    A set is a function of its members: (#{"the"} "the") returns "the", and nil otherwise.
  2. 2. What does (fnil inc 0) do?

    fnil replaces a nil first argument with the default, so update works on missing keys.
  3. 3. Why sort-by [(- cnt) word]?

    Vectors compare element by element: the negated count puts large counts first, and the word decides ties.
  4. 4. Which is the right way to print each result row?

    map and for are lazy and may never run the side effect; doseq executes it eagerly.

Summary

  • Model the program as a pipeline of pure functions; keep file reading and printing at the edges.
  • re-seq tokenizes, sets filter stop words, frequencies counts, sort-by with a vector key ranks deterministically.
  • doseq for output, slurp and line-seq for input, ->> for readable flow.
  • Extend with bigrams, files, transducers and pmap with merge-with.

Next: Project: Event-Sourced Bank Accounts