Skip to content
Tutorials Center
My learning

Topics

Lesson 134 of 147 Β· Projects

Project: Word Frequency Analyzer

Build a word counter in Go in four stages - reading stdin, Unicode-aware tokenizing, counting with maps, ranking the top N, and a concurrent worker-pool version checked against the simple one.

Counting words sounds trivial until you try to do it properly. Is CafΓ© the same word as cafΓ©? What about don't written with a typewriter apostrophe and don’t written with a curly one? Where does a word end when text contains ZΓΌrich! or naΓ―ve? or the number 2026? Real text is messy, and handling it well is a good exercise in Go's strings, runes, maps and sorting. Counting is also a perfect problem for learning a pattern you will meet again and again: split the work across goroutines, let each one build its own result, and merge the results at the end.

In this project you build a word-frequency analyzer in four stages, and each stage is a complete program. In stage 1 you read text from standard input, split it into words with the unicode package and count them in a map. In stage 2 you clean up the words, skip stop words, rank the results and print a report with bars. In stage 3 you turn the counter into a concurrent version with a pool of workers. In stage 4 you compare three concurrent designs and check that every one of them produces exactly the same answer.

You'll use maps, strings and runes, the unicode package, sorting, goroutines, channels and the sync package.

The sample text arrives through the Input box, which the program reads as standard input, exactly as if you had typed go run . < essay.txt in a terminal. Edit the text and press Run to try your own. The input contains accented letters, curly apostrophes and numbers on purpose.

The plan

Stage What you build Main Go ideas
1 Read stdin, split into words, count in a map bufio.Scanner, strings.FieldsFunc, unicode.IsLetter, maps
2 Normalizing, stop words, top N, bars, statistics Types with methods, slices.SortFunc, cmp.Or, utf8.RuneCountInString
3 A worker pool that counts in parallel Channels, sync.WaitGroup.Go, per-worker maps, merging
4 Three concurrent designs, compared sync.Mutex, contention, result equality with maps.Equal

Stage 1: read, split, count

The heart of the program is three steps. A scanner delivers the input line by line. A function breaks a line into words. A map from word to count records how often each word appears.

How should you decide where a word starts and ends? Splitting on spaces is not enough, because punctuation sticks to words (dog. and dog would be different keys). Instead, say what a word character is and treat everything else as a separator. strings.FieldsFunc(s, f) splits s around every rune for which f returns true. Our function says: a separator is any rune that is not a letter, a digit or an apostrophe. unicode.IsLetter knows about letters in every script, so ΓΌ, Γ― and Γ© count as letters, and so would я or ζ—₯.

Before splitting, the line is lowercased with strings.ToLower, so The and the become the same word.

Maps have no order, so printing a map directly would give a different order on every run. The program collects the keys, sorts them with slices.Sorted(maps.Keys(counts)) and prints in that order. Go compares strings byte by byte, so the order is not a dictionary order: 'naΓ―ve' (which starts with an apostrophe) comes before the digits and letters, and an accented letter sorts after every plain ASCII letter.

Go Stage 1: tokenize and count

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The program works, but look at the output with a critical eye. 'naΓ―ve' (with the quotes) and naΓ―ve are counted separately, don’t keeps its curly apostrophe, café’s is its own word, and a number such as 2026 is counted as a word. Words like the and is dominate the list. These are not bugs in the code so much as questions about what the program should do, and stage 2 answers them.

Some details that are easy to miss:

  • counts[w]++ works on a missing key. Reading a missing key from a map gives the zero value, 0, so the increment starts from zero. You never need an "if not present" check for counters. See maps.
  • FieldsFunc never returns empty strings. Several separators in a row (a comma and a space) produce no empty words, unlike strings.Split.
  • Always check scanner.Err() after the loop. Scan returns false both at the end of input and on a read error, and only Err tells them apart. Also remember that a Scanner rejects lines longer than 64 KB unless you enlarge its buffer with scanner.Buffer.
  • Runes, not bytes. ToLower and IsLetter work on Unicode code points, which is why ZΓΌrich stays in one piece. If you looped over the string with for i := 0; i < len(s); i++ you would see single bytes and cut ΓΌ in half. See strings and runes.

Stage 2: clean words, rank them, print a report

Stage 2 turns the script into a small library. A Counter type owns the map and the rules, and its methods are the only code that touches the data:

  • Add(word) normalizes the word, drops it if it is a stop word, too short or a pure number, and counts it otherwise.
  • AddText(line) tokenizes a line and calls Add for each piece.
  • Top(n) returns the n most frequent words as a slice of Entry values.

Normalizing means: replace the curly apostrophe ’ with a straight one, and trim apostrophes from both ends, so 'naΓ―ve' becomes naΓ―ve. Stop words are very common words such as the, and and is that would otherwise fill the top of every ranking. They live in a map[string]struct{}, which is Go's idiomatic set: the empty struct takes no space, and a lookup is _, skip := stop[word].

Ranking needs a total order, or the output would change between runs. Sorting by count alone leaves ties in an arbitrary order. The comparison function therefore uses cmp.Or to chain two keys: higher count first, then alphabetical.

The report prints each word with its count, its share of all counted words, and a bar scaled to the most common word. To align the columns, the program first finds the widest word. It measures in runes with utf8.RuneCountInString, because len("cafΓ©") is 5 bytes while the word is 4 characters wide. The fmt package also counts width in runes, so %-*s (the * takes the width from an argument) pads cafΓ© and programs to the same visual width.

Go Stage 2: a Counter, stop words and a ranked report

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The first report is dominated by the, which is exactly why stop words exist. In the second report, go leads, followed by a tie of words that appear twice, in alphabetical order. The curly apostrophe in don’t is now stored as a straight one (don't), and 'naΓ―ve' has joined naΓ―ve, which is why naΓ―ve now has a count of 2. The statistics at the bottom show how many words each filter removed: "found" counts every token, "counted" only those that survived.

Design points worth remembering:

  • Ties need a rule. Without the alphabetical tie-breaker, two words with the same count could swap places between runs, because Top starts from a map. cmp.Or(a, b) returns the first non-zero comparison, which makes multi-key sorting read like a sentence.
  • min(n, len(entries)) (a built-in since Go 1.21) prevents slicing past the end when fewer than n words exist.
  • The Options struct has useful zero values. NewCounter(Options{}) means "no stop words, no minimum length". Looking up a key in a nil map is legal and reports "absent", so the zero Options just works. See constructors and options.
  • strings.ContainsFunc (Go 1.21) asks "does any rune satisfy this test?", so isNumber says: no rune is a non-digit.

Real text-analysis tools use more advanced ideas, such as stemming (programs and program are one word), possessives, and language-specific stop word lists. Add them behind the same Add method and nothing else in the program changes. Keeping all word rules in one place is the reason Counter exists.

Stage 3: counting in parallel with a worker pool

For a short essay, one goroutine is plenty. For a gigabyte of logs, you would like to use every CPU core. Counting is an ideal candidate for parallelism because the work on one line does not depend on any other line. The plan is a worker pool with local results:

  1. The main goroutine cuts the input into batches of lines and sends them on a channel.
  2. A fixed number of workers each take batches from that channel and count words in their own private Counter. No two workers ever touch the same map, so no locking is needed.
  3. When the batches run out, each worker sends its finished Counter on a results channel.
  4. The main goroutine merges all the partial counters into one.

This design is sometimes called map-reduce in miniature: the workers map (count), and the final merge reduces. Its great strength is that workers share nothing while they work. Its cost is a merge step at the end, which is cheap compared with the counting.

The code uses sync.WaitGroup.Go (Go 1.25) to start the workers, slices.Chunk (Go 1.23) to cut the lines into batches, and a small goroutine whose only job is to close the results channel after all workers are done. To make the input big enough to matter, the program repeats the sample text 400 times. The result is compared with the simple sequential count, using maps.Equal.

Go Stage 3: a worker pool that counts in parallel

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Every worker count gives exactly the sequential answer. That is the test that matters for concurrent code: same input, same output, whatever the number of workers or the order in which they finish. (The order in which workers finish is not deterministic, but merging is addition, and addition does not care about order. That is why the program is correct.)

Understanding the channel choreography is the key to the pattern:

  • batches is closed by its producer. Only the sender closes a channel. When the producer goroutine has sent every batch it closes batches, which ends each worker's for batch := range batches loop.
  • results is closed by a goroutine that waits. Many workers send on results, so no single worker may close it. A helper goroutine calls wg.Wait() and then closes it, which ends the main goroutine's for local := range results. This "wait and close" helper is a standard idiom.
  • The workers send exactly one value each. results is unbuffered, so a worker blocks until main receives its counter. That is fine because main receives in a loop.
  • No data race. Each Counter is written by one goroutine only; ownership moves to main when it is sent through the channel. slices.Chunk slices share memory with corpus, but workers only read them.

Do not take any speedup for granted. On a machine with one CPU, which is what this site's runner has, four workers cannot run faster than one. Even with many cores, tiny inputs lose time to goroutine and channel overhead. Measure with a benchmark on your own data before adding concurrency: see benchmarks and fuzzing.

Stage 4: three designs, one answer

The private-map design is not the only way to count concurrently. A simpler-looking alternative is one shared map protected by a mutex: every worker locks, increments, unlocks. It has no merge step, but every single word waits for the lock, so workers spend their time queueing. A middle road is sharding: split the map into several independent maps, each with its own mutex, and pick the shard by hashing the word. Workers then collide only when two words hash to the same shard.

Stage 4 implements all three designs behind one function signature, func(lines []string, workers int) map[string]int, and checks that every one of them returns the same map. It is a nice example of testing concurrent code by comparing against a trivially correct sequential reference.

The sharded design uses hash/maphash, which gives a fast, well-distributed hash for strings. Each program run uses a random seed, so the shard assignment changes between runs, but the final totals cannot.

Go Stage 4: local maps, one big lock and shards

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

All three designs are correct. They differ only in how much time workers spend waiting, and that depends on the data and the hardware, so measure before you choose. As a rule of thumb:

  • Private maps win when each unit of work is large (a batch of lines) and the merge is cheap compared with the counting. Workers never wait on each other while counting.
  • One big lock is simplest to reason about and perfectly adequate when the work between lock operations is large. Counting words is the opposite case: the work per lock is a single increment, so the lock is the bottleneck.
  • Sharded locks reduce contention when you must have a shared structure that many goroutines update continuously, such as a cache. See concurrency patterns and Redis and in-process caches.

The most important line in the whole stage is the comparison with the sequential want. It would catch a missing lock, a lost update or a bad merge in all three designs. The run go test -race goes further and also catches unsynchronized access that happens to produce the right numbers by luck. On your own machine you would write the comparison as a test and run it with the race detector, as described in testing concurrent code:

Terminal
go test -race -run TestCountersAgree ./...
go test -bench=Count -benchmem

Ideas for extending the analyzer

  • Read a file or several files given as arguments, and process each file in its own goroutine.
  • Count bigrams (pairs of adjacent words) with a map[[2]string]int; arrays are comparable, so they can be map keys.
  • Replace the stop word list with one loaded from a file with //go:embed.
  • Stream huge inputs: use bufio.Reader with a big buffer or scanner.Buffer(make([]byte, 1<<20), 1<<20) and never hold all lines in memory.
  • Print a histogram of word lengths, or the words that appear exactly once ("hapax legomena").
  • Output JSON with encoding/json so another program can use the result.

Check your understanding

Quick check Β· 5 questions

of 5 answered You scored / 5 Β· perfect!

  1. 1. Why does the tokenizer use strings.FieldsFunc with unicode.IsLetter instead of strings.Fields?

    strings.Fields splits on white space only, so dog. and dog would differ. FieldsFunc lets you define separators yourself, and unicode.IsLetter recognizes letters of every script.
  2. 2. What does counts[w]++ do when w is not yet in the map?

    Reading a missing key yields the zero value of the element type, so increment works without a presence check.
  3. 3. Two words both appear 3 times. Why does Top still print them in the same order on every run?

    Map iteration order is random, so without a tie-breaker the result could vary. cmp.Or chains the count comparison with an alphabetical one, giving a total order.
  4. 4. In the worker pool, who closes the results channel, and why not the workers?

    Only one party may close a channel, and only after all sends are done. A goroutine that waits for the WaitGroup knows exactly when that moment arrives.
  5. 5. Why is the 'private maps' design usually better than 'one big lock' for counting words?

    With a shared lock every word increment queues behind the others. Private maps avoid contention; the cost is a merge that is small compared with counting.

Summary

  • Decide what a word is by defining the separator test and using strings.FieldsFunc with the unicode package; normalize case and apostrophes before counting.
  • A map from word to count needs no presence check: a missing key reads as 0. Collect and sort keys for stable output, because map order is random.
  • Wrap rules and data in a type (Counter) so that tokenizing, filtering and ranking live in one place. Use a total order (cmp.Or with a tie-breaker) when ranking.
  • In a worker pool, share nothing while working: each worker fills a private result, and the main goroutine merges. The producer closes the input channel, and a "wait and close" goroutine closes the results channel.
  • Check concurrent code against a simple sequential reference, and run it with go test -race. Measure before assuming that concurrency makes a program faster.
  • Width in fmt verbs counts runes, but len counts bytes: use utf8.RuneCountInString when you need the display length of text.

Next: Project: Bank Account System