Linux Command Reference ยท Text Processing
Linux uniq command
Reference for the Linux uniq command: collapse adjacent duplicate lines, count them, show only duplicates or only unique lines, and skip fields or characters.
uniq filters out adjacent repeated lines. It reads a file or standard input, and for each run of identical lines writes just one. With options it can prefix each line with how many times it occurred, print only the lines that were repeated, or print only those that never repeated.
The key word is adjacent: uniq does not remember lines it saw earlier, which is why it is almost always used after sort. The classic frequency table is sort | uniq -c | sort -rn.
uniq [OPTION]... [INPUT [OUTPUT]]
Try it
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.
Syntax
uniq [-cdDiu] [-f N] [-s N] [-w N] [-z] [INPUT [OUTPUT]]
| Option | Description |
|---|---|
-c |
Prefix each output line with the number of occurrences in its group. |
-d |
Print one copy of each group that has more than one line. |
-D |
Print all lines of every repeated group. |
-u |
Print only lines that are not repeated. |
-i |
Ignore case when comparing. |
-f N |
Skip the first N fields (blank-separated) when comparing. |
-s N |
Skip the first N characters when comparing. |
-w N |
Compare no more than N characters per line. |
-z |
Lines end in NUL instead of newline. |
--group |
Separate groups with an empty line; accepts separate, prepend, append, both. |
INPUT, OUTPUT |
Input file (default standard input) and output file (default standard output). |
Returns: 0 on success, non-zero on error. It does not signal whether duplicates were found.
Usage notes
- Sort first.
uniqonly compares each line to the previous one, soa b astaysa b a. For deduplicating unsorted data without reordering, useawk '!seen[$0]++'. sort -uis the shortcut when you only want distinct lines and no counts.uniqearns its place when you need-c,-dor-u.-cpads the count to seven columns followed by a space. Pipe throughsed 's/^ *//'orawk '{ print $1 }'if you need to parse it.-fand-scombine: fields are skipped first, then characters. Field separators are runs of blanks, and the blanks before a field count as part of it.- Whole-line options:
-dand-ulook at the same groups, so every group is printed by exactly one of-dor-u. - The output file argument is a rarely used second positional argument. A mistaken extra file name silently overwrites that file, so quote carefully when scripting.
- Locale: the comparison is bytewise equality (or case-folded with
-i), so a locale never changes which lines are equal, but it does change howsortorders them. - Large inputs work in constant memory because
uniqkeeps only one line at a time.
Related
sort,comm,cut,awk- Lessons: sort, uniq and cut, Text pipelines