Skip to content
Tutorials Center
My learning

Topics

Lesson 74 of 134 · Standard Library

Text Processing

Learn practical Ruby text processing - splitting and cleaning text, counting words, parsing log files, transforming with tr and gsub, and printing aligned reports.

A huge share of everyday programming is text work: reading a log, cleaning up a pasted list, counting words, turning "name,score" lines into a report. Ruby grew up on this kind of job. Its string methods, regular expressions and the Enumerable methods you already know combine into short programs that would take pages in other languages.

This lesson is a workshop of patterns. Each section shows one technique on a small piece of sample text that lives inside the program, so everything runs in your browser. The next lessons (Files and IO, StringIO and Streams) show how the same code reads real files; the text-handling part does not change.

Think in lines, words and fields

Almost every text task has the same shape: break the text into lines, break each line into fields, convert the fields to proper types, and then summarize. lines, each_line and split do the breaking.

Ruby Lines and fields

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

<<~TEXT is a squiggly heredoc: it strips the common indentation, so you can indent the text to match your code. lines keeps the trailing newline of every line unless you pass chomp: true. each_line does the same one line at a time without building the whole array, which matters for big text.

Study the split results, because they surprise people:

  • split(",") drops trailing empty fields but keeps empty ones in the middle. Pass -1 as the limit to keep them all, which is what you want when column positions matter.
  • split with no argument splits on any run of whitespace and ignores leading whitespace, so it is the right tool for words.
  • A numeric limit such as 2 stops after that many pieces and leaves the rest of the text intact.

Use String#to_i and to_f for quick scripts: they quietly return 0 for garbage. Use Integer("12") and Float("0.25") when you want an exception on bad input. The difference decides whether a typo in your data is noticed or silently counted as zero.

Cleaning text

Real text is messy: extra spaces, odd case, stray punctuation. A small toolbox handles most of it.

Ruby The cleaning toolbox

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

tr translates characters one-for-one (and understands ranges like A-Z), delete removes the listed characters, squeeze collapses runs of the same character, and count counts characters from a set. These character-set methods are simple and fast, and they are the first choice when you do not need a full regex. strip removes whitespace at both ends; lstrip and rstrip handle one side. delete_prefix and delete_suffix remove a literal piece, with no pattern surprises.

Remember that string methods without a bang return a new string. The original raw is unchanged after all of these calls.

Counting words

Counting is the classic text exercise, and Ruby makes it two lines: extract the words, then tally.

Ruby Word frequency

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

scan(/[a-z']+/) extracts runs of letters and apostrophes, which is how punctuation gets thrown away without a separate cleaning step. Sorting by [-n, word] sorts by count descending and then alphabetically, so ties come out in a stable, predictable order. Always add a tie-breaker when you show the "top N", or the result may change when the input order changes.

Parsing a log file

Logs are the most common real-world text. Each line has a regular structure, so a regex with named groups turns lines into hashes. Lines that do not match are collected, not ignored: bad data should be visible.

Ruby Parsing a log into records

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Four ideas make this robust. The \A...\z anchors force the whole line to follow the format. Named groups give you readable access to the pieces. The inner scan extracts key=value pairs where a value is either a quoted string or a run of non-space characters. And the rejected array records the line number of anything unparseable, so you can report it instead of silently losing data. Once the lines are hashes, ordinary select, group_by and tally answer questions.

Splitting by structure instead of regex

When the format is simple, partition and split are clearer than a regex. partition splits at the first separator and always returns three parts, which makes it perfect for key: value lines.

Ruby partition for key-value lines

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

When the separator is missing, partition returns the whole string followed by two empty strings, so checking whether the second element is empty tells you the line was not a pair.

Wrapping and truncating

Terminal output and email text need lines of a limited width. Ruby has no built-in word wrap for plain text (Rails adds one), but a short regex does the job.

Ruby Word wrap and truncation

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The wrap pattern grabs up to width characters followed by whitespace or the end of the string, and puts a newline after each chunk. A word longer than the width is not split, it simply overflows. For a very different approach, each_slice on the word list chops by word count instead of character count.

Aligned reports

A report must line up in columns. ljust, rjust and center pad strings, and format (also %) controls numbers precisely.

Ruby Building a formatted report

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Compute the column width from the data instead of hard-coding it, so a long name does not break the layout. In format strings, %-10s pads a string to 10 characters on the right, %8.2f right-aligns a float with two decimals in a field of 8, %05d zero-pads an integer, and %% prints a literal percent sign. The last line is an alternative to the lookaround comma trick from the regex lesson.

Building text efficiently

Repeated += on a string creates a new string every time. For big output, append with << or collect pieces in an array and join once.

Ruby Appending and joining

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The last two blocks show why this matters beyond speed. t += "d" makes a new string and leaves s alone. v << "d" mutates the one string that both u and v point to, so both change. Know which you are doing. +"" creates an unfrozen empty string even when the file uses # frozen_string_literal: true.

Encodings and non-ASCII text

Ruby strings are UTF-8 by default and index by character, not byte, so Unicode text mostly just works. Be careful with methods that count bytes.

Ruby Characters versus bytes

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

length counts characters (10) while bytesize counts bytes (12), because ï and é take two bytes each in UTF-8. A byte sequence that is not valid UTF-8, like "caf\xE9", reports valid_encoding? as false. Operations such as split or a regex match on an invalid string raise ArgumentError with the message invalid byte sequence in UTF-8, so check input from the outside world, or clean it with String#scrub, which replaces each bad byte (here with ?).

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. What does "a,b,,d,,".split(",") return?

    split removes trailing empty fields but keeps the empty one in the middle. Pass -1 as the limit to keep every empty field.
  2. 2. Which call gives you the first separator split into exactly three parts, even when the separator is missing?

    partition always returns [before, separator, after]; when the separator is missing, the last two parts are empty strings.
  3. 3. How do you count how often each word appears in an array called words?

    tally returns a hash from each value to the number of times it appears.
  4. 4. Why is s.tr("A-Z", "a-z") different from s.downcase?

    tr translates only the characters in the ranges you gave. downcase understands full Unicode case rules, so it also lowercases letters like À.
  5. 5. After t = s; t += "d", what is true about s?

    += is a = a + b, which builds a new string and rebinds t. With << the same string would have been mutated for both names.

Summary

  • Break text into lines (lines, each_line(chomp: true)), then fields (split, partition, scan), then convert and summarize.
  • split drops trailing empty fields unless you pass -1; split with no argument is the word splitter.
  • strip, squeeze, tr, delete, count, delete_prefix and delete_suffix are fast, pattern-free cleaners.
  • scan plus tally, group_by and max_by turn text into statistics; add a tie-breaker when sorting.
  • Parse structured lines with an anchored regex with named groups, and keep the lines that fail.
  • ljust, rjust, center and format align reports; compute widths from the data.
  • Build big strings with << or join, and remember length counts characters while bytesize counts bytes.

Next: Files and IO