Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 95 ยท Working with Text

Hex and Binary Inspection: xxd, od, hexdump, strings

Inspect files that are not text on Linux: read hex dumps with xxd, od and hexdump, find readable text with strings, identify formats with file, and compare binaries with cmp.

Most of this track treats files as lines of text. But a lot of files are not text: images, archives, compiled programs, databases, and text files that have been damaged by a wrong encoding or a stray Windows line ending. cat on such a file prints garbage and may even mess up your terminal. To see what is really inside, you need tools that show bytes.

This lesson covers the standard toolbox: file to identify a format, strings to pull out readable text, od, xxd and hexdump to print bytes in hexadecimal, cmp to compare binary files, and dd to cut out a slice. The sandbox has od, file, strings, cmp, dd and gcc, but not xxd or hexdump; those two are shown as read-only examples, and the runnable ones use od, which does the same job. It builds on Viewing Text, Streams and Disk Usage and File Information.

Bytes, hex and why magic numbers matter

A file is a sequence of bytes, each a number from 0 to 255. Hexadecimal (base 16) writes one byte as exactly two digits (00 to ff), which makes it the natural way to read binary data. Many formats start with a fixed magic number that identifies them:

First bytes (hex) As text Format
7f 45 4c 46 .ELF Linux executable or library
89 50 4e 47 0d 0a 1a 0a .PNG.... PNG image
ff d8 ff JPEG image
50 4b 03 04 PK.. ZIP archive (also .jar, .docx)
1f 8b gzip compressed data
25 50 44 46 %PDF PDF document
23 21 #! Script with an interpreter line

The file command reads those magic numbers (and more) to tell you what a file is, regardless of its extension. The extension is only a hint; the bytes are the truth.

file: what is this?

CONSOLE
$ file photo.jpg report.pdf archive.tar.gz script.sh /bin/ls
photo.jpg:      JPEG image data, JFIF standard 1.01 ...
report.pdf:     PDF document, version 1.7
archive.tar.gz: gzip compressed data, from Unix ...
script.sh:      Bourne-Again shell script, ASCII text executable
/bin/ls:        ELF 64-bit LSB pie executable, x86-64 ...

file -i (or --mime-type) prints a MIME type like image/png, handy in scripts. file -b drops the file name. For text, file also reports the encoding and line-ending style, for example ASCII text, with CRLF line terminators, which is the quickest way to spot a file from Windows.

Bash Identify files by their bytes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Your exact file wording may differ a little between versions, but the format names (PNG, gzip, shell script, CRLF) will match.

od: the portable dump

od ("octal dump") is part of POSIX and exists everywhere, including tiny systems without xxd. Its power is in the options:

Option Meaning
-A x / -A d / -A n Address column in hex, decimal, or none
-t x1 Show each byte as hex (x2/x4 group 2 or 4 bytes)
-t c Show bytes as characters, with escapes like \n and \0
-t d1 / -t u1 Bytes as signed / unsigned decimal
-N n Read only the first n bytes
-j n Skip the first n bytes
-v Do not collapse repeated lines into a *
Bash Reading bytes with od

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

In the z form of -t (-tx1z) od appends a printable-character column on the right, much like xxd does. The -c view is the best way to tell a Unix newline (\n) from a Windows one (\r \n) and to spot invisible characters such as a tab or a byte-order mark.

xxd and hexdump (read-only here)

xxd ships with Vim and is the friendliest dump: offset, hex bytes in pairs, and the text on the right.

CONSOLE
$ xxd small.txt
00000000: 4869 210a                                Hi!.

$ xxd -l 16 /bin/ls               # first 16 bytes only
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............

$ xxd -p small.txt                # plain hex, no layout
4869210a

$ echo 4869210a | xxd -r -p       # reverse: hex back to bytes
Hi!

$ xxd -b small.txt                # binary digits (bits)
$ xxd -s 8 -l 4 file.bin          # skip 8 bytes, show 4

xxd -r is the reason people love it: you can dump a file, edit the hex text in an editor, and rebuild the binary file. hexdump -C (from util-linux/bsdmainutils) gives the similar "canonical" layout:

CONSOLE
$ hexdump -C small.txt
00000000  48 69 21 0a                                       |Hi!.|
00000004

All three show the same facts; use whichever is installed. In scripts, prefer od (always there), and xxd -p when you need a pure hex string, or od -An -tx1 | tr -d ' \n' to get the same without xxd.

strings: the readable parts

strings scans a file for sequences of printable characters (default: at least 4 in a row) and prints them. It is the fastest way to learn what a binary contains: error messages, file paths, version numbers, URLs, the names of libraries it uses.

CONSOLE
$ strings -n 8 /bin/ls | head        # at least 8 characters
$ strings -a binary | grep -i version
$ strings -t x binary | head          # also print each string's offset in hex

Be careful: strings found in a file are data, not proof of what the program does. Still, it is a standard first step when analysing an unknown file or checking whether a secret was accidentally compiled into a build. The next example compiles a tiny C program, looks at its magic number and finds its message with strings.

Bash Inspect a compiled program

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Slicing and comparing

dd copies bytes with exact control. Use bs=1 skip=N count=M to cut a slice by offset, or bigger block sizes for speed. Do not point of= at a device unless you are certain of the name: dd has earned the nickname "disk destroyer".

cmp compares two files byte by byte and reports the first difference, which is what you want for binary files. diff is for text; see Comparing Files: diff and patch.

Bash Compare and edit binary data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Note that cmp -l prints byte numbers in decimal (starting at 1) and the values in octal, a historical quirk. Checksums such as sha256sum are the right tool to check that a large file arrived intact.

Random bytes: /dev/urandom, shuf and $RANDOM

Sometimes you need random data: a password, a token, a test file. The kernel supplies cryptographically strong random bytes through /dev/urandom (and /dev/random, which behaves the same on current kernels). Because it is an endless stream of arbitrary bytes, filter it with tr -dc ("delete the complement"), which keeps only the characters you allow, and cut it with head -c. Set LC_ALL=C so tr treats the input as raw bytes.

Terminal
LC_ALL=C tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 16; echo      # a 16-character password
LC_ALL=C tr -dc 'A-Za-z0-9!@#%^&*_+' < /dev/urandom | head -c 24; echo   # with symbols
head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n'; echo          # 32 hex characters

$RANDOM is a Bash variable that gives a number from 0 to 32767 each time you read it. It is fine for a shuffled test or a die roll ($(( RANDOM % 6 + 1 ))) but not for secrets, because it comes from a simple generator. shuf -i 1-100 -n 1 picks a random number in a range and shuf file shuffles lines.

Bash Generate and check random values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

A workflow for unknown files

  1. file for the type, then ls -l for the size.
  2. head -c 64 file | od -An -c (or xxd | head) to see the header.
  3. strings -n 6 file | less for hints (names, URLs, messages).
  4. If it looks like text but behaves oddly, od -c reveals \r, tabs, a leading byte-order mark (ef bb bf) or stray NUL bytes.
  5. Compare with a known-good copy using cmp, and verify downloads with sha256sum -c.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A file ends in .txt, but od -An -tx1 -N4 prints 7f 45 4c 46. What is it?

    7f 45 4c 46 is the ELF magic number (DEL, E, L, F). The extension is only a hint; the first bytes identify the format.
  2. 2. Which command prints a file's bytes as two-digit hex values without an address column?

    -An suppresses the address and -tx1 shows one byte per hex value. -c shows characters, and strings and cmp do other jobs.
  3. 3. How do you tell a Windows (CRLF) line ending from a Unix one with od?

    CRLF is two bytes: carriage return (0x0d) and line feed (0x0a). od -c makes the \r visible, and file also reports it as 'with CRLF line terminators'.
  4. 4. What does strings -n 8 program do?

    strings only reads the file. -n sets the minimum length of a printable run to report.
  5. 5. Which tool compares two binary files and tells you the position of the first differing byte?

    cmp compares bytes and reports the first difference (differ: byte 5, line 1). diff works on lines of text.

Summary

  • Files are bytes; magic numbers at the start (ELF 7f 45 4c 46, PNG, gzip 1f 8b, ZIP PK) identify a format, and file reads them for you.
  • od is always available: -An -tx1 for hex, -c for characters and escapes, -N and -j to limit or skip bytes. xxd and hexdump -C give friendlier layouts, and xxd -r turns hex back into a file.
  • strings finds readable text in binaries; strings -t d also gives offsets.
  • cmp compares binary files byte by byte, dd cuts or patches by offset, and sha256sum verifies whole files.
  • od -c reveals hidden characters in text files: carriage returns, tabs, byte-order marks and NUL bytes.

Next: Standard Streams and Redirection