Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 37 ยท Projects and Practice

Project: Word Frequency Counter

Build a word frequency counter in Zig step by step using slices, tokenizing, a StringHashMap, an arena for owned keys, sorting and tests, with a runnable Go version of the same design.

This project ties together slices, strings, allocators, hash maps, sorting and testing. You will write a function that takes text and returns the most common words with their counts. We keep the I/O at the edges (the main function) so the core logic is easy to test, which is the usual Zig shape.

The plan

  1. Tokenize: split text into lowercase words, ignoring punctuation.
  2. Count: put words into a StringHashMap(u32). Keys are slices, so they must live as long as the map: use an arena.
  3. Sort: copy the entries into a list and sort by count (descending), then word (ascending) for a stable result.
  4. Report: print the top N.
  5. Test the counting function with std.testing.allocator.
flowchart LR
  T["text"] --> A["tokenize<br/>lowercase words"]
  A --> M["StringHashMap<br/>word -> count"]
  M --> L["ArrayList of entries"]
  L --> S["sort by count desc"]
  S --> R["print top N"]

Caption: the data flows through five small steps, each easy to test on its own.

Step 1: the data types

ZIG
const std = @import("std");

pub const Entry = struct {
    word: []const u8,
    count: u32,
};

fn lessThan(_: void, a: Entry, b: Entry) bool {
    if (a.count != b.count) return a.count > b.count; // bigger counts first
    return std.mem.lessThan(u8, a.word, b.word);      // then alphabetical
}

std.mem.sort wants a comparison function with a context parameter (here unused, so it is void). Breaking ties by word makes the output deterministic, since hash map order is not.

Step 2: counting

ZIG
pub fn countWords(arena: std.mem.Allocator, text: []const u8) !std.StringHashMap(u32) {
    var counts = std.StringHashMap(u32).init(arena);

    var start: ?usize = null;
    for (text, 0..) |ch, i| {
        if (std.ascii.isAlphabetic(ch) or ch == '\'') {
            if (start == null) start = i;
        } else if (start) |s| {
            try addWord(arena, &counts, text[s..i]);
            start = null;
        }
    }
    if (start) |s| try addWord(arena, &counts, text[s..]);
    return counts;
}

fn addWord(arena: std.mem.Allocator, counts: *std.StringHashMap(u32), raw: []const u8) !void {
    const lower = try std.ascii.allocLowerString(arena, raw);
    const gop = try counts.getOrPut(lower);
    if (gop.found_existing) {
        gop.value_ptr.* += 1;
    } else {
        gop.value_ptr.* = 1;
    }
}

Points to notice:

  • We walk the text once, tracking where the current word started. A word is a run of letters or apostrophes.
  • allocLowerString makes an owned, lowercase copy. The map keeps that slice as a key, so it must outlive the map. It was allocated from the same arena, so everything is freed in one deinit.
  • A tiny inefficiency: when the word already exists, the lowercase copy is wasted (the arena holds it anyway). That is the price of simplicity. A more careful version would look up with a stack buffer first and duplicate only when new; try it as an exercise.
  • The function takes an arena, not a general allocator, which documents the ownership rule: do not free individual strings, drop the arena instead.

Step 3: ranking

ZIG
pub fn topWords(
    gpa: std.mem.Allocator,
    counts: std.StringHashMap(u32),
    n: usize,
) ![]Entry {
    var entries = try std.ArrayList(Entry).initCapacity(gpa, counts.count());
    errdefer entries.deinit(gpa);

    var it = counts.iterator();
    while (it.next()) |kv| {
        entries.appendAssumeCapacity(.{ .word = kv.key_ptr.*, .count = kv.value_ptr.* });
    }
    std.mem.sort(Entry, entries.items, {}, lessThan);

    const keep = @min(n, entries.items.len);
    entries.shrinkRetainingCapacity(keep);
    return entries.toOwnedSlice(gpa);
}

We reserve the exact capacity first, so appendAssumeCapacity cannot fail. The errdefer frees the list if anything later fails, and toOwnedSlice hands the caller an exactly-sized slice (the caller frees it with gpa.free). The Entry.word slices still point into the arena, so the arena must outlive the result.

Step 4: main

ZIG
pub fn main() !void {
    var gpa_state: std.heap.DebugAllocator(.{}) = .init;
    defer _ = gpa_state.deinit();
    const gpa = gpa_state.allocator();

    var arena_state: std.heap.ArenaAllocator = .init(gpa);
    defer arena_state.deinit();
    const arena = arena_state.allocator();

    const text =
        \\The quick brown fox jumps over the lazy dog.
        \\The dog barks; the fox runs away.
    ;

    var counts = try countWords(arena, text);
    // the map's own table is allocated from the arena as well: no deinit needed

    const top = try topWords(gpa, counts, 3);
    defer gpa.free(top);

    for (top) |e| {
        std.debug.print("{s}: {d}\n", .{ e.word, e.count });
    }
}

Expected output: the: 4, then dog: 2, then fox: 2 (ties are alphabetical). The \\ lines are Zig's multiline string literal.

Step 5: tests

ZIG
test "counts are case-insensitive and ignore punctuation" {
    var arena_state: std.heap.ArenaAllocator = .init(std.testing.allocator);
    defer arena_state.deinit();
    const arena = arena_state.allocator();

    const counts = try countWords(arena, "Hi, hi! HI? ho");
    try std.testing.expectEqual(@as(?u32, 3), counts.get("hi"));
    try std.testing.expectEqual(@as(?u32, 1), counts.get("ho"));
    try std.testing.expectEqual(@as(usize, 2), counts.count());
}

test "topWords orders by count then word" {
    var arena_state: std.heap.ArenaAllocator = .init(std.testing.allocator);
    defer arena_state.deinit();
    const counts = try countWords(arena_state.allocator(), "b a b a c");

    const top = try topWords(std.testing.allocator, counts, 2);
    defer std.testing.allocator.free(top);

    try std.testing.expectEqualStrings("a", top[0].word);
    try std.testing.expectEqualStrings("b", top[1].word);
}

The testing allocator sits under the arena, so any leak of the arena's own blocks would fail the test. The top slice is allocated directly from it and freed with defer.

Where to take it next

  • Read the text from a file or stdin (see the I/O documentation of your Zig version) and pass the bytes to countWords.
  • Count with a stack buffer to avoid allocating for repeated words.
  • Add a --top N option parsed from std.process.args.
  • Treat the text as UTF-8 and handle non-ASCII letters.
  • Make countWords generic over a "normalizer" function, using the ideas from Generic functions with comptime.

The same design, runnable

Zig cannot run here, so this is the same algorithm in Go: tokenize, count, sort by count then word, print the top three. The structure (and the expected the: 4) matches the Zig version.

Go Word frequency in Go (same design as the Zig project)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is an arena allocator a good fit for the word map's keys?

    StringHashMap stores key slices, not copies. Allocating the lowercase words from an arena keeps them alive until the arena is deinitialized.
  2. 2. Why break count ties alphabetically?

    Without a secondary key equal counts appear in arbitrary order, which makes output and tests flaky.
  3. 3. What does toOwnedSlice do?

    The list gives up ownership of its memory, and the caller becomes responsible for freeing the returned slice.
  4. 4. Why keep file I/O out of countWords?

    Passing the data in makes the logic testable without files.

Summary

  • Split the problem into tokenize, count, sort, report.
  • Own map keys with an arena; sort entries copied into a list with a deterministic comparator.
  • Use errdefer for partial setups and toOwnedSlice to return results.
  • Test with std.testing.allocator on top of an arena.

Next: Project: a stack-based calculator VM