Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 95 ยท Working with Text

sort, uniq, cut, tr and wc

Master sort, uniq, cut, tr, wc and join: numeric and multi-key sorting, frequency tables, top-N lists, locale effects and stable sorts, with runnable examples.

Some questions about text have the same shape again and again. Which value occurs most often? What are the ten biggest files? How many lines, words or unique users are in this export? Linux answers them with a small team of tools: sort puts lines in order, uniq collapses neighbours that are equal, cut and tr reshape the data, wc counts, and join merges two tables on a shared key. None of them is clever alone. Chained with pipes they replace a surprising amount of spreadsheet and scripting work.

You met cut and tr in Viewing and Slicing Text. This lesson focuses on the tools that bring order and numbers: sort, uniq, wc and join, and on the recipes that combine all of them. By the end you will build frequency tables and "top N" lists from memory, and you will know why sort | uniq -c | sort -rn is one of the most used commands in all of Unix.

sort: putting lines in order

sort reads lines, orders them and prints the result. By default it compares lines as text, character by character, using the rules of your locale. That default surprises everybody once, so let us see it right away.

Bash sort: text order versus numeric order

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The sort -c check prints a message such as sort: nums.txt:3: disorder: 100 on stderr and exits with status 1. Text order explains 1, 10, 100, 9: the first character decides, 1 is smaller than 9, and so on. As soon as the data are numbers, add -n. The most useful options are:

Option Meaning
-n numeric comparison (leading number of the key, whole numbers and decimals)
-h human-numeric: understands 2K, 15M, 3G (GNU)
-r reverse the result
-u unique: drop later lines whose key equals an earlier one
-k N sort by field N (or a range such as -k2,2) instead of the whole line
-t X field separator is the character X (default: blank-to-nonblank transitions)
-s stable: keep the original order of lines that compare equal
-f fold case (ignore upper/lower case)
-V version sort: file2 before file10 (GNU)
-o file write the result to a file (safe even if it is also the input)

Sorting by a column with -k and -t

Real data have columns. -t, says "fields are separated by commas" and -k3 says "compare starting at field 3". There is one trap you must know: -k3 means "from field 3 to the end of the line". Almost always you want -k3,3: only field 3. Without the ,3 the later fields take part in the comparison, and they can change the order of lines with equal third fields.

You can chain keys: -k3,3nr -k2,2 means "field 3 as a number, descending; ties broken by field 2 as text". The letters after a key (n, r, h, f) apply to that key only.

Bash sort -k, -t and multiple keys

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Look at the -u result: with -k2,2 -u GNU sort keeps one line for every distinct department. Which line it keeps among equals is not something to rely on; if it matters, sort first with explicit tie-breakers or use awk.

Stable sorts and why order of equal lines matters

When two lines compare equal on the key, the default sort falls back to comparing the entire line as a tie-breaker (the "last resort" comparison). That makes its output fully deterministic, but it also means equal-key lines are not kept in their original order. With -s the last-resort comparison is switched off, and equal-key lines stay in input order. That is a stable sort. It lets you sort in layers: sort by the secondary key first, then stable-sort by the primary key.

Bash Stable sort keeps the original order of ties

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Compare the first two blocks. For priority 2 the default prints refactor before update logo (alphabetical), and for priority 3 it prints add tests before write docs; -s instead keeps update logo before refactor and write docs before add tests, exactly as in the file. The third block gets alphabetical ties on purpose: the first sort orders by text, and the stable second sort keeps that order inside each priority.

Locale: why sort order differs between machines

The order of letters depends on the locale (LC_COLLATE). In the plain C locale, sort compares raw byte values: all uppercase letters come before all lowercase ones. In a locale such as en_US.UTF-8 it follows dictionary rules where case and punctuation are mostly ignored on the first pass. Scripts that must behave the same on every machine set LC_ALL=C. Our runner sets LC_ALL=C.UTF-8 for you, which also compares code points, so the examples here are reproducible.

Bash LC_ALL=C: byte order versus dictionary rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice that sort -fu printed Apple and banana: among lines that compare equal, which one -u keeps is not something to depend on. A fast tip: LC_ALL=C sort is also much faster on large files, because comparing bytes is cheaper than applying locale rules. It is a standard trick in big data pipelines. The cost is that non-ASCII text is ordered by byte value, not by language rules.

sort -n on a locale with a comma as the decimal separator (for example German) treats 1,5 as one and a half and 1.5 as text. If a script sorts decimals, set LC_ALL=C so the dot is always the decimal point.

uniq: collapsing neighbours

uniq removes adjacent duplicate lines. The word adjacent is the whole story: uniq does not remember lines it saw earlier, so it only works properly on sorted input. That is why you nearly always write sort | uniq. (When you do not need counts, sort -u does both steps in one.)

Option Meaning
-c prefix each line with the number of repeats
-d print only lines that occur more than once (one copy each)
-D print every copy of duplicated lines
-u print only lines that occur exactly once
-i ignore case when comparing
-f N / -s N skip the first N fields / characters when comparing
Bash uniq: adjacent only, plus -c, -d and -u

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The classic mistake is uniq file on unsorted data and concluding "there are no duplicates". The first block above shows red still twice. When the input must keep its original order, awk '!seen[$0]++' removes duplicates without sorting; you will meet that in awk.

The frequency-table idiom

Put the pieces together and you get the most important recipe of the lesson:

TEXT
produce items  |  sort  |  uniq -c  |  sort -rn  |  head
   one per line    group     count      biggest first   top N
  1. Get one item per line (with cut, tr, grep -o, ...).
  2. sort so equal items are neighbours.
  3. uniq -c turns each group into count item.
  4. sort -rn orders by that count, largest first.
  5. head -n N keeps the top N.
Bash Word frequency: the sort | uniq -c | sort -rn recipe

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

uniq -c right-aligns the counts in a field of width 7, followed by one space and the item. Downstream you can parse it with awk '{print $1, $2}' or strip the padding with sed 's/^ *//'.

Look at the words with count 1. After sort -rn they appear as wind, wave, sky, sea, sand: reverse alphabetical order. The -r flag reverses the last-resort whole-line comparison as well, so ties come out backwards. If you want ties in alphabetical order, say so explicitly with -k1,1nr -k2,2 as in the last command.

wc: counting lines, words and bytes

wc (word count) prints counts for files or standard input:

  • -l lines (the number of newline characters)
  • -w words (runs of non-blank characters)
  • -c bytes, -m characters
  • -L length of the longest line

Without options you get lines, words and bytes, in that order. With input from a pipe there is no file name, so the output is just the numbers, which is what you want for $(...).

Bash wc: lines, words, bytes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last example is a famous off-by-one: wc -l counts newline characters, so a final line without a trailing newline is not counted. That is another reason to end text files with a newline. Also note that wc -c counts bytes: รฉ is two bytes in UTF-8, so use -m when you want characters.

cut and tr in the pipeline

cut chooses columns and tr translates, deletes or squeezes characters (details in Viewing and Slicing Text). In frequency tables they are the first stage: they produce the "one item per line" stream.

Bash Counting values in a column

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The "total" line uses command substitution: paste -sd+ produces 30+45+12+80+20+25, and $(( ... )) evaluates it. For sums grouped by country you will want awk (awk), which has variables and arrays.

sort -h and sizes

sort -n cannot compare 900K with 2M. The -h option reads the suffixes (K, M, G, ...) that ls -lh and du -h print.

Bash sort -h for human-readable sizes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

You will use du -h | sort -h all the time to find what fills a disk, see Disk Usage.

join: merging two tables on a key

join is a tiny relational join. It reads two files that are sorted on the join field and prints a line for every pair of lines with equal keys, key first, then the remaining columns of file 1, then of file 2. Useful options: -t, separator, -1 N / -2 N key field in file 1/2, -a 1 also print unmatched lines of file 1 (a left join), -e X filler for missing fields (with -o to fix the output columns).

Bash join: combine two sorted files

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

If the files are not sorted on the join field, join complains (join: ... is not sorted) or silently skips matches. Sort both inputs with the same key and the same locale; LC_ALL=C for both is the safe way. For complex joins people reach for awk, a database, or the sqlite3 command, but for two sorted lists join is fast and has no memory limit.

Top-N recipes

Here is a small gallery that you can adapt. Each one is a pipeline of the tools above.

Bash Three top-N recipes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Common pitfalls

  • uniq without sort. Only neighbours are merged. Sort first, or use sort -u.
  • sort -k3 instead of -k3,3. The key then reaches the end of the line.
  • Missing -n. Numbers sorted as text: 10 before 9.
  • Leading blanks matter. uniq -c pads with spaces; sorting such output with plain sort compares the blanks. Use sort -rn (numeric mode ignores leading blanks) or -b.
  • Different locales in different steps. A join after a sort in another locale fails. Export LC_ALL=C at the top of a script.
  • Trusting -u to choose a line. sort -u -k2,2 keeps a line per key, not necessarily the one you want.
  • Large files. sort spills to temporary files automatically, so it handles files bigger than memory. You can set the buffer with -S 1G and the temp directory with -T /big/disk.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does sort print 100 before 9 for the numbers 9, 100 and 1000?

    Without -n sort compares character by character, so '1' sorts before '9'. Adding -n makes it compare the numeric value.
  2. 2. You have an unsorted file with repeated lines. What does uniq file print?

    uniq only compares neighbouring lines, so non-adjacent duplicates survive. Use sort | uniq or sort -u to remove all duplicates.
  3. 3. Which pipeline lists the three most frequent words, one per line, from a file with one word per line?

    Sort groups equal words, uniq -c counts each group, sort -rn puts the biggest counts first and head keeps the first three.
  4. 4. What is the difference between sort -k3 and sort -k3,3?

    A key without an end position runs to the end of the line, so later fields influence the order. -k3,3 limits the key to field 3.
  5. 5. What does printf 'a\nb' | wc -l print?

    wc -l counts newline characters. The final line b has no trailing newline, so only one newline is counted.

Summary

  • sort orders lines as text by default; use -n for numbers, -h for sizes like 2M, -r to reverse, -V for version numbers and -u to drop duplicates.
  • Sort by column with -t and -k N,N; chain keys for tie-breaking and use -s for a stable sort.
  • Set LC_ALL=C for byte order, reproducible results and speed.
  • uniq merges only adjacent lines: sort first. -c counts, -d shows duplicates, -u shows singletons.
  • sort | uniq -c | sort -rn | head is the frequency-table and top-N idiom.
  • wc -l counts newline characters; wc -w, -c, -m, -L count words, bytes, characters and the longest line.
  • join merges two files that are sorted on a common key; cut, tr and paste prepare the data.

Next: diff, comm and patch