Skip to content
Tutorials Center
My learning

Topics

Lesson 83 of 134 · Standard Library

Digest, Base64 and Encoding

Learn Ruby's Digest hashes (SHA256, HMAC), Base64 and URL-safe Base64, hex, pack and unpack, and string encodings - force_encoding vs encode, scrub, and common byte-level errors.

Text on your screen is a comfortable illusion. Underneath, every string is a sequence of bytes, and a handful of everyday jobs live at that level: fingerprinting data with a hash, squeezing binary data into a form that survives email or JSON, reading numbers out of a binary file header, or converting between character encodings such as UTF-8 and Latin-1. Ruby covers all of them with small tools: Digest for hashes, Base64 for text-safe binary, pack and unpack for byte layouts, and the Encoding machinery built into every string.

This lesson shows each tool in turn and then the part that trips up professionals: the difference between changing the label on a string's bytes (force_encoding) and converting the bytes (encode). On the way you will meet the errors these tools raise, with the real messages, so they feel familiar when they show up in your own code.

digest ships with Ruby and loads with require 'digest'. base64 is a bundled gem since Ruby 3.4: it installs together with Ruby, so require 'base64' works in plain scripts and on our runner, but in a real project that uses Bundler you must add gem "base64" to your Gemfile (see RubyGems and Bundler). If you only need the basic encoding, [data].pack("m0") does the same job without any library, as shown below.

Digests: fingerprints of data

A digest (or hash) function turns any amount of input into a short, fixed-size fingerprint. The same input always gives the same fingerprint, a tiny change gives a completely different one, and you cannot realistically go backwards from the fingerprint to the input. Ruby's Digest library offers SHA256, SHA384, SHA512, SHA1, MD5 and others.

Ruby hexdigest and friends

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

There are three flavours of each class method. hexdigest returns the fingerprint as a string of hexadecimal characters (64 for SHA-256, because 32 bytes need two hex characters each), digest returns the raw bytes in a binary string, and base64digest returns the bytes as Base64. Changing one letter of the input changed 63 of the 64 hex characters: that is the avalanche effect, and it is why digests are good for detecting even tiny modifications.

Hashing big or streaming data

You do not have to hold everything in memory. Create a digest object and feed it pieces with << (or update). The result is the same as hashing the whole thing at once.

Ruby Incremental digests and file checksums

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Digest::SHA256.file(path) reads a file in chunks and returns the digest object, which is how download tools produce a checksum you can compare with a published one. The last line shows that a digest only ever sees bytes: the UTF-8 string "héllo" and the same bytes written out by hand give the same fingerprint. Hash the same text in a different encoding and you get a different digest, so agree on UTF-8.

Giving something that is not a string fails with an ordinary type error:

Ruby Digests need strings (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The run ends with main.rb:3:in 'Digest::Class.digest': no implicit conversion of Integer into String (TypeError). Convert first with 123.to_s.

What digests are for, and what they are not for

  • Integrity checks. Compare checksums of a file before and after a transfer.
  • Cache keys and deduplication. Digest::SHA256.hexdigest(content) makes a stable name for content.
  • Change detection. Rebuild something only when the digest of its inputs changes.

What they are not for: storing passwords. Fast hashes such as SHA-256 let an attacker try billions of guesses per second. Passwords need a deliberately slow, salted algorithm such as bcrypt, scrypt or Argon2, from a gem built for that job. And MD5 and SHA1 are broken for security purposes (attackers can craft collisions); you will still see them for non-security checksums, but pick SHA256 for anything new.

HMAC: a digest with a secret key

To prove that a message came from someone who knows a shared secret (webhook signatures, signed cookies), mix a key into the hash. That is an HMAC, available from the openssl library that ships with Ruby.

Ruby HMAC and constant-time comparison

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Use OpenSSL.secure_compare rather than == for secrets and signatures. A plain == stops at the first differing character, so the time it takes leaks how many leading characters were right, which an attacker can measure. The secure version always takes the same time.

Base64: binary data as text

Some channels only carry text: email bodies, JSON, URLs, HTTP headers. Base64 turns arbitrary bytes into a safe alphabet of letters, digits, + and / (and = padding), at the cost of making the data about one third bigger.

Ruby Encoding and decoding Base64

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Prefer strict_encode64: it produces one unbroken line with no newline. The plain encode64 inserts a newline every 60 characters and at the end, which is what old email formats wanted but rarely what you want in JSON. Decoded data comes back as a binary string (encoding ASCII-8BIT, which inspect displays as BINARY (ASCII-8BIT)), because Base64 cannot know whether the bytes were text. Tell Ruby what they are with force_encoding("UTF-8") if you know it was text.

URL-safe Base64

The characters + and / have special meaning in URLs. The URL-safe variant uses - and _ instead:

Ruby urlsafe_encode64 and padding

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

JWTs and many tokens use the URL-safe form without the = padding (padding: false); urlsafe_decode64 accepts input both with and without padding. The regular decode64 is forgiving to a fault: it silently skips characters that are not Base64, so garbage in gives garbage out. strict_decode64 raises ArgumentError with the message invalid base64 on bad input, which is what you want when the data comes from outside.

Base64 is an encoding, not encryption. Anyone can decode it, so it never protects a secret. It only makes bytes safe to carry.

Hex, pack and unpack

Hexadecimal is another way to write bytes as text, two characters per byte. pack and unpack convert between Ruby values and binary strings using a compact format language, which is how you read file headers, network protocols and image formats.

Ruby pack and unpack basics

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The most useful directives: C (one unsigned byte), n/N (16/32-bit big-endian), v/V (16/32-bit little-endian), s, l, q with > or < for explicit byte order, a/A (string padded with NUL or spaces), H (hex), m (Base64), U (UTF-8 code point), and * for "as many as there are". unpack1 returns the first value instead of an array. Integer#to_s(16) and String#hex handle the common single-number cases.

Encodings: bytes plus a label

Every Ruby string carries an encoding: the label that says how to interpret its bytes as characters. Source code and most data are UTF-8, in which one character takes one to four bytes.

Ruby Characters, bytes and encodings

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

"héllo" has 5 characters but 6 bytes, because é takes two bytes (195, 169, or C3 A9). length counts characters, bytesize counts bytes: use the right one for the job (database column limits and HTTP Content-Length are in bytes). Note that 233.chr without an encoding gives the single byte "\xE9", which is not a valid UTF-8 character; pass Encoding::UTF_8 to get "é".

force_encoding versus encode

These two look similar and do opposite things:

  • force_encoding(enc) changes only the label. The bytes stay the same. Use it when you know what the bytes mean but Ruby guessed wrong (data read in binary mode, Base64 output).
  • encode(enc) converts: it changes the bytes so the same characters are represented in the new encoding. Use it to translate text between systems.
Ruby Relabelling versus converting

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The classic bug is calling force_encoding when you needed encode (or the other way around). If a string looks wrong and valid_encoding? is false, the bytes and the label disagree. Decide which one is correct: the bytes (then fix the label) or the label (then the data is damaged).

The errors, and how to handle them

Conversions can fail because a character does not exist in the target encoding, or because the bytes are not valid in the source encoding. Ruby tells you with specific exception classes.

Ruby Conversion errors (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

This stops with main.rb:1:in 'String#encode': U+00E9 from UTF-8 to US-ASCII (Encoding::UndefinedConversionError): the character é cannot be written in ASCII. You decide what should happen instead, using options:

Ruby Replace, fall back, scrub

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

undef: :replace swaps characters that do not exist in the target for ? (or your replace: text), fallback: maps specific characters, and scrub repairs a string that contains invalid bytes. The other three errors show typical failures with damaged data: encoding with an invalid byte (InvalidByteSequenceError), a regexp match on an invalid string (ArgumentError: invalid byte sequence in UTF-8), and joining two strings whose non-ASCII characters belong to different encodings (Encoding::CompatibilityError). The cure for all three is the same: normalise to UTF-8 at the edge of your program.

Unicode normalisation

One visible character can have more than one byte representation. é is either a single code point or e plus a combining accent. They look identical and compare unequal.

Ruby unicode_normalize

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Normalise user input with unicode_normalize(:nfc) before comparing, hashing or using it as a key. The last line decomposes accents, then removes the combining marks (\p{Mn}) to get a plain ASCII-looking resume, a common trick for building URL slugs (see Regular Expressions).

A checklist for byte-level code

  • Hash and sign bytes: choose UTF-8 for text and be consistent on both sides.
  • Use strict_encode64/strict_decode64 unless you have a reason not to; use urlsafe_* for URLs and tokens.
  • Compare secrets with OpenSSL.secure_compare, never ==.
  • Convert to UTF-8 on input, work in UTF-8 inside, convert back on output. Read files in binary mode ("rb") when they are not text.
  • When a string looks like é mojibake or raises an encoding error, ask: which is wrong, the bytes or the label?

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. Which method returns a SHA-256 fingerprint as a 64-character hexadecimal string?

    hexdigest gives two hex characters per byte, so 32 bytes become 64 characters. digest returns the raw 32 bytes and base64digest returns a Base64 string.
  2. 2. What does Base64 provide?

    Base64 is only an encoding. Anyone can decode it, and the output is about a third bigger than the input.
  3. 3. A string has the wrong encoding label, but its bytes are correct UTF-8. Which call fixes it?

    force_encoding changes only the label and leaves the bytes alone. encode would try to convert the bytes from the old label, which is wrong when the label was the mistake.
  4. 4. Why use OpenSSL.secure_compare instead of == for comparing signatures?

    A plain == returns at the first difference, and the timing can reveal how much of a guess was correct. secure_compare avoids that.
  5. 5. What does "héllo".bytesize return, and why?

    UTF-8 uses a variable number of bytes per character. h, l, l and o take one byte each and é takes two, so length is 5 but bytesize is 6.

Summary

  • Digest::SHA256.hexdigest(data) fingerprints bytes; feed large inputs with << or Digest::SHA256.file(path). Use SHA-256 or better, never MD5/SHA1 for security, and never plain hashes for passwords.
  • OpenSSL::HMAC.hexdigest signs a message with a secret; compare signatures with OpenSSL.secure_compare.
  • Base64 (a bundled gem since Ruby 3.4) carries bytes as text: prefer strict_encode64, use urlsafe_encode64 for URLs, and remember it is not encryption.
  • pack and unpack convert between Ruby values and byte layouts: C, n/N, v/V, H, m, U.
  • A string is bytes plus an encoding label. force_encoding relabels, encode converts; handle failures with undef: :replace, fallback: and scrub.
  • Normalise Unicode text with unicode_normalize(:nfc) before comparing it.

Next: URI and Networking Basics