Linux Command Reference ยท Concepts
The Unix philosophy
The Unix philosophy explained with working shell examples: small programs that do one job, text as the common interface and pipelines that combine them.
The Unix philosophy is a style of building software that explains why the Linux command line feels the way it does. Instead of one huge program that does everything, you get many small programs that each do one thing well, and a way to connect them. The shell supplies the connection: the pipe.
The ideas were written down by the people who built early Unix, and they still shape how commands are designed today. They are not laws, but when a tool breaks them it is usually harder to script, and when it follows them you can combine it with tools its author never imagined.
cat words.txt | sort | uniq -c | sort -rn | head -n 3
Try it
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.
The core ideas
Do one thing well
wc counts, sort sorts, uniq collapses duplicates. None of them knows about the others. A tool with a narrow job is easier to understand, test, document and replace. When you need something new, ask first which existing tools already cover pieces of it.
Text is the universal interface
Most commands read lines of text and write lines of text. Because the format is shared, the output of any tool can be the input of any other, and you can inspect any intermediate result with your eyes. This is the reason grep, cut, sort and awk combine so freely. It also means that data should be easy to parse: one record per line, fields separated by a predictable character.
Compose with pipes and redirection
A pipe connects the standard output of one process to the standard input of the next, and all stages run at the same time, passing data as it is produced. Redirection connects a stream to a file. Together with the three standard streams they form the glue. See stdin, stdout and stderr and Pipes.
Everything is a file
Devices, sockets, pipes and even kernel information appear in the filesystem (/dev, /proc, /sys), so the same tools (cat, echo, grep) work on all of them. Reading CPU details is cat /proc/cpuinfo; discarding output is > /dev/null. See The filesystem hierarchy.
Be quiet on success, loud on failure
A well-behaved program prints nothing when everything worked, writes errors to standard error, and sets its exit status (0 for success). Scripts rely on this: if grep -q ... works because grep says nothing and reports through its exit code. See Exit codes.
Build prototypes early, automate repeated work
Write the first version as a shell pipeline, run it, and refine. If you type the same pipeline three times, make it a function or script. Only when the shell becomes awkward (complex data, speed, structure) move to a general-purpose language.
Usage notes
- Filters are the model citizens. A filter reads standard input when no file is named and writes standard output;
sort,grep,sedandtrall behave like this. Design your own scripts the same way so they plug into pipelines. - Keep stdout for data and stderr for messages. Progress and warnings sent to stdout would corrupt the data flowing to the next tool.
- Separate policy from mechanism. Tools provide mechanisms (sort, find); you supply the policy (which key, which directory) through options and arguments rather than hard-coded behavior.
- Beware of delimiters. Text pipelines break when data contains the separator, such as spaces in file names. Use NUL-separated streams (
find -print0 | xargs -0) for file lists. - Exit status of a pipeline is that of the last command unless you set
pipefail. See Bash strict mode. - Not everything fits. Nested data (JSON, XML) is awkward for line-oriented tools; use a dedicated parser such as
jqfor those, and keep the rest of the pipeline. - Small does not mean slow. Pipeline stages run concurrently and share the kernel's pipe buffers, so many pipelines process gigabytes with little memory.
- Combine rather than extend. Before adding a flag to a tool or writing a new one, check whether
tee,xargs,pasteorcommalready solve the problem.
Related
sort,uniq,tr,xargs,tee- stdin, stdout and stderr, grep vs sed vs awk, Useful one-liners
- Lessons: Pipes, Text pipelines, Introduction