Skip to content
Tutorials Center
My learning

Topics

Lesson 65 of 66 ยท Projects and Practice

Project: A Tiny Shell with Pipes

Build a tiny command shell simulator in Python that parses command lines, runs commands as streaming filters connected by pipes, handles redirection and &&, and learn the fork, exec, pipe and dup2 behind a real shell.

A shell is the program that turns what you type into running processes. It looks like magic, but its main loop is simple: read a line, parse it, create processes, connect their input and output, wait for them, and print a prompt again. Pipes (a | b) and redirection (a > file) are the shell's most famous tricks, and they rest on a handful of system calls that you now know: fork, exec, pipe, dup2 and wait.

A browser cannot start real processes, so this project simulates a shell in Python. Each program is a generator that reads lines from its input and yields lines to its output, which models exactly how a pipeline works: data flows as a stream, and no stage needs to finish before the next one starts. The project ends with the real system-call version, shown as read-only C so you can see how the simulation maps onto a real kernel.

How a real shell runs cat f | sort > out

sequenceDiagram
    participant S as Shell
    participant C1 as Child 1 (cat)
    participant C2 as Child 2 (sort)
    S->>S: pipe() creates a read end and a write end
    S->>C1: fork()
    C1->>C1: dup2(write end, stdout), exec cat
    S->>C2: fork()
    C2->>C2: dup2(read end, stdin), open out, dup2(out, stdout), exec sort
    S->>S: close both pipe ends, wait for both children
    C1-->>C2: bytes flow through the kernel pipe buffer

The shell builds the plumbing before the programs start. The programs never know they are connected to a pipe.

The key facts:

  • A pipe is a kernel buffer with two file descriptors: one to write, one to read. When the buffer is full the writer blocks; when empty the reader blocks (the producer-consumer problem again).
  • fork clones the shell. The child inherits all open descriptors.
  • dup2(old, new) makes descriptor new refer to the same open file as old. That is how a child's file descriptor 1 (standard output) is redirected into a pipe or a file.
  • exec replaces the child's program while keeping its descriptors, so the new program inherits the plumbing.
  • The shell waits for the children and reports the exit status (0 means success). a && b runs b only if a returned 0.

Step 1: parsing

A shell must split a line into stages at |, split each stage into words (respecting quotes), and pull out redirections (>, >>, <). Python's shlex.split handles the quoting rules.

Python Step 1: parse pipelines and redirections

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each stage becomes a small record: the program and its arguments, and where its input and output go.

Step 2: pipelines as chained generators

Now the engine. Every program is a Python generator function. To connect stage k to stage k + 1, we simply pass the first generator as the lines argument of the second. Nothing runs until the last stage asks for data, and then lines flow through the whole chain one at a time, which is what makes pipes streaming: cat hugefile | head -1 stops reading after one line, just as the real one would.

The shell also keeps a toy file system (a dict), does $VAR expansion, hands out pretend process IDs, honors > and >>, and stops an && chain when a command fails. Unknown commands return status 127, as in a real shell.

Python Step 2: a working shell simulator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Study the outputs:

  • cat fruits.txt | sort | uniq -c | sort -r | head -2 is the classic "most common word" idiom. Five processes, each one tiny, combine to answer a question that no single program does. This is the Unix philosophy: small programs that do one thing and communicate through plain text streams.
  • cat ... > a.txt && cat a.txt | wc shows redirection writing a file, then a second pipeline reading it back.
  • cat nothing.txt && echo never printed stops at the first failure, and frobnicate reports "command not found".

Step 3: the real system calls

This is a read-only C sketch (error handling removed) of what a real shell does for cat fruits.txt | sort. It maps one to one onto the simulation above.

C
#include <unistd.h>
#include <sys/wait.h>

int main(void) {
    int fd[2];
    pipe(fd);                          /* fd[0] = read end, fd[1] = write end */

    if (fork() == 0) {                 /* child 1: cat */
        dup2(fd[1], STDOUT_FILENO);    /* stdout now goes into the pipe */
        close(fd[0]); close(fd[1]);
        execlp("cat", "cat", "fruits.txt", (char *)NULL);
        _exit(127);                    /* only reached if exec failed */
    }
    if (fork() == 0) {                 /* child 2: sort */
        dup2(fd[0], STDIN_FILENO);     /* stdin now comes from the pipe */
        close(fd[0]); close(fd[1]);
        execlp("sort", "sort", (char *)NULL);
        _exit(127);
    }
    close(fd[0]); close(fd[1]);        /* the shell must close its copies, or sort never sees EOF */
    int status;
    wait(&status);                     /* reap both children */
    wait(&status);
    return 0;
}

The close calls matter. A reader sees end-of-file only when every write end of the pipe has been closed. If the shell kept its own copy open, sort would wait forever. This is one of the most common bugs in shell-writing assignments.

Step 4: job control (ideas)

Real shells do more:

  • Background jobs (cmd &): the shell skips wait and keeps prompting; it reaps children later when it receives SIGCHLD.
  • Foreground and background process groups, so Ctrl-C (SIGINT) reaches only the foreground pipeline.
  • Built-in commands such as cd, exit and export must run inside the shell process, because a child cannot change its parent's working directory or environment.
  • Signals, $?, globbing, history and tab-completion.
  • Exit status: the value wait reports, checked by &&, || and if.

Extensions

  1. Add || (run the right side only if the left side fails) and ;.
  2. Add cd and pwd with a working directory, and make relative paths resolve against it (see path resolution).
  3. Add tail, tr, cut -d, sed s/a/b/, and tee.
  4. Add a background job syntax & by running the pipeline later and listing it with jobs.
  5. Replace the generators with real multiprocessing pipes on your own machine and measure the overhead.
  6. Implement a one-line command history with !!.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does dup2(fd[1], STDOUT_FILENO) do in the child of a pipeline?

    After dup2, file descriptor 1 refers to the pipe's write end, so everything the program prints goes into the pipe.
  2. 2. Why must the shell close its own copies of the pipe's descriptors?

    EOF is delivered only after every write end is closed. A leftover copy in the shell keeps the reader waiting forever.
  3. 3. Why must cd be a shell built-in rather than an external program?

    A directory change affects only the process that makes it. A separate program would change its own directory, not the shell's.
  4. 4. In a && b, when does b run?

    && is a success-conditional: it checks the exit status of the left command.
  5. 5. Why is the generator model a good simulation of pipes?

    A pipe also moves a stream rather than a finished result, so stages overlap and head can end the chain early.

Summary

  • A shell parses a line, builds pipes and redirections with pipe and dup2, creates children with fork, runs programs with exec and waits for them.
  • Pipelines are streams: each stage reads from the previous one, and no stage needs to finish before the next one starts.
  • Exit statuses drive && and ||; built-ins like cd must run inside the shell itself.
  • Forgetting to close pipe ends is the classic bug that makes pipelines hang.

Next: Operating Systems Practice Problems