Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 37 ยท Projects and Practice

Project: Text Statistics

Build a text analyzer in F#: split text into words, count frequencies with Seq.countBy, rank with sortBy, compute averages and longest words, read files lazily, and print a report with pipelines.

This project is a classic for a reason: it shows off pipelines, collections and records on a problem everybody understands. You will write a small program that reads text and reports how many words it has, which words are most common, the longest word and the average word length. Every step is a short function, and the final program is one pipeline.

The data

We want the result as a record, so the analysis is separated from printing:

FSHARP
type Stats =
    { Characters: int
      Words: int
      UniqueWords: int
      AverageLength: float
      Longest: string
      Top: (string * int) list }

Step 1: words

Split on anything that is not a letter or apostrophe, lower-case it and drop empty pieces:

FSHARP
open System

let words (text: string) : string list =
    text.Split([| ' '; '\n'; '\r'; '\t'; '.'; ','; ';'; ':'; '!'; '?'; '"'; '('; ')'; '-' |],
               StringSplitOptions.RemoveEmptyEntries)
    |> Array.map (fun w -> w.Trim('\'').ToLowerInvariant())
    |> Array.filter (fun w -> w <> "")
    |> List.ofArray

You could also use a regular expression (Regex.Matches(text, "[\\p{L}']+")) for full Unicode letters. See the string functions in Collections and Sequences for Seq and List basics.

Step 2: counting

List.countBy groups equal items and counts them in one call:

FSHARP
let frequencies (ws: string list) =
    ws
    |> List.countBy id           // (word, count) pairs in first-seen order
    |> List.sortByDescending snd // most common first, stable for ties

id is the identity function ("group by the item itself"). List.sortByDescending is stable, so words with the same count keep first-seen order, which keeps the output deterministic.

C# C# stand-in: counting words with LINQ

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Step 3: the report

FSHARP
let analyse (text: string) : Stats =
    let ws = words text
    let freq = frequencies ws
    { Characters = text.Length
      Words = ws.Length
      UniqueWords = freq.Length
      AverageLength =
        if ws.IsEmpty then 0.0 else ws |> List.averageBy (fun w -> float w.Length)
      Longest =
        if ws.IsEmpty then "" else ws |> List.maxBy String.length
      Top = freq |> List.truncate 5 }

List.averageBy throws on an empty list, which is why the guard is there; the empty case is a good test. List.maxBy String.length picks the first longest word, because maxBy returns the first maximum it finds.

flowchart LR
    T["text"] --> W["words"]
    W --> F["countBy + sort"]
    W --> A["averageBy / maxBy"]
    F --> S["Stats record"]
    A --> S
    S --> P["print report"]

One pass of splitting feeds several small analyses that merge into one record.

Step 4: printing

FSHARP
let print (s: Stats) =
    printfn "Characters : %d" s.Characters
    printfn "Words      : %d (%d unique)" s.Words s.UniqueWords
    printfn "Avg length : %.2f" s.AverageLength
    printfn "Longest    : %s" s.Longest
    printfn "Top words:"
    for word, count in s.Top do
        printfn "  %-10s %3d" word count

"The cat saw the dog. The dog saw the cat, and the bird saw them both!"
|> analyse
|> print

The %-10s format pads a string to a width of 10, left-aligned. Expected output (ties keep first-seen order):

TEXT
Characters : 69
Words      : 16 (8 unique)
Avg length : 3.19
Longest    : bird
Top words:
  the          5
  saw          3
  cat          2
  dog          2
  and          1

Step 5: reading a file lazily

For a big file do not load everything at once. File.ReadLines returns a lazy seq<string>, so counting words streams through the file in constant memory:

FSHARP
let countFile (path: string) =
    System.IO.File.ReadLines path
    |> Seq.collect (fun line -> words line)       // flatten the words of every line
    |> Seq.countBy id
    |> Seq.sortByDescending snd
    |> Seq.truncate 10
    |> Seq.toList

[<EntryPoint>]
let main argv =
    match argv with
    | [| path |] when System.IO.File.Exists path ->
        for word, count in countFile path do
            printfn "%-15s %d" word count
        0
    | _ ->
        eprintfn "usage: wordcount <file>"
        1

Sequences are evaluated on demand, so Seq.truncate 10 after sorting still has to read the whole file once (sorting needs everything), but it never holds the file text in memory, only the word counts.

C# C# stand-in: lazy line-by-line processing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Ideas to extend it

  1. Ignore common stop words (the, and) by filtering against a Set<string>.
  2. Count sentences, or the longest sentence, with Regex.Split.
  3. Plot a histogram of word lengths with String.replicate n "#".
  4. Compare two files with Set.intersect and Set.difference on their unique words.
  5. Use Array.Parallel for many files and merge the Map<string,int> results with Map.fold.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does List.countBy id do to a list of words?

    countBy groups items by the key function (here the item itself) and counts each group.
  2. 2. Why does analyse check for an empty list before averageBy?

    The average of nothing is undefined, so the empty case needs its own result.
  3. 3. What is the advantage of File.ReadLines over File.ReadAllLines for big files?

    ReadLines streams the file as a sequence, keeping memory use low.
  4. 4. Which format specifier pads a string to width 10, left-aligned?

    The minus flag left-aligns within the given width.

Summary

  • Keep analysis (a Stats record) apart from printing, and build each step as a small function.
  • countBy, sortByDescending, averageBy, maxBy and truncate do the heavy lifting.
  • Guard the empty case wherever a function throws on empty input.
  • File.ReadLines plus Seq functions processes large files lazily.

Next: Practice Exercises