Skip to content
Tutorials Center
My learning

Topics

Lesson 48 of 95 ยท Processes and the System

Jobs and Background Processes

Run commands in the background with & and $!, collect results with wait, limit concurrency, use timeout, and learn jobs, fg, bg, nohup and disown in Bash.

A shell normally runs one command at a time: you type it, it runs, you get the prompt back. But many tasks do not need you to wait. A download can happen while you edit a file, ten independent checks can run side by side, and a server should keep going after you log out. The shell's answer to all of these is the job: a command (or pipeline) that the shell starts and keeps track of, in the foreground or in the background.

In this lesson you will learn how to start background jobs with &, how to find their process IDs with $!, how to wait for them and collect their exit statuses, how to run work in parallel without overloading the machine, how to stop a command that runs too long with timeout, and what jobs, fg, bg, nohup and disown do in an interactive terminal. It builds on Processes and Signals and trap.

Foreground and background

A command that you run normally is a foreground job: the shell starts it, then waits until it ends. Adding & at the end starts it as a background job instead. The shell does not wait; it immediately moves on to the next line.

TEXT
  foreground:   shell โ”€โ”€startโ”€โ”€โ–ถ [ sleep 2 ] โ”€โ”€endsโ”€โ”€โ–ถ shell continues
  background:   shell โ”€โ”€startโ”€โ”€โ–ถ [ sleep 2 ]  ........ runs on its own
                  โ”‚
                  โ””โ”€ continues at once with the next command
Bash The shell does not wait for a background job

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The first echo ran immediately, long before the sleep ended. The builtin wait then blocks until all background children of the shell have finished. Without it, a script would simply reach its end and exit while the job might still be running.

$! and wait: tracking a job

When you start a background job, Bash stores its process ID (PID) in the special variable $!. Save it at once, because the next background job overwrites it:

Terminal
long_task &
pid=$!

wait PID blocks until that one process ends and then returns its exit status as the status of wait itself. This is how you find out whether a background job succeeded.

Bash Collecting the exit status of a background job

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice that you can wait for the jobs in any order, even if they finished in another order: the shell remembers the status of a finished child until you ask for it. A plain wait with no argument waits for all children but always returns 0, so it throws the statuses away. If you care about failures, wait for each PID and check each result.

A background job is a child of the shell that started it. wait only works for your own children. You cannot wait for an unrelated process that someone else started; the shell answers "pid N is not a child of this shell".

Running work in parallel

Two background jobs started one after the other run at the same time. This turns a slow sequential loop into a quick one.

Bash Sequential versus parallel

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

In the parallel part each job writes into its own file, and we print the files in a fixed order afterwards. That is a good habit: output from several processes written to one terminal interleaves unpredictably, so give every job its own output file or collect results by index.

Collecting every status

A common pattern is to remember all PIDs in an array and then wait for each:

Bash Running checks in parallel and counting failures

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The array pids and the array targets have matching indexes, so index i tells you which target a PID belongs to. See Arrays for the "${!pids[@]}" syntax.

Limiting concurrency

Starting 1,000 jobs at once can exhaust memory, file handles or the process limit. A simple throttle: start jobs, and when the number of running jobs reaches your limit, wait for one of them to finish. Bash 4.3+ has wait -n for exactly this: it returns as soon as any one background job ends.

Bash At most 2 jobs at a time with wait -n

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

jobs -rp prints the PIDs of the jobs that are still running. At no time are more than two job processes alive. For simple "run this command for each input" tasks, xargs -P does the same with less code; see xargs.

timeout: do not wait forever

Sometimes a command might hang, such as a network call or a script waiting for input. The GNU command timeout DURATION COMMAND runs the command and kills it if it is still running after the duration. It exits with status 124 when it had to kill the command; otherwise it passes through the command's own status.

Bash Giving a command a time budget

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The Killed line in the error stream is Bash itself reporting that the command died from a signal. By default timeout sends SIGTERM, which a program can catch or ignore. --signal=KILL (or -s KILL) chooses another signal. For a stubborn program, timeout --kill-after=5 30 cmd sends TERM after 30 seconds and, if the command is still alive 5 seconds later, KILL. The duration is a number of seconds, optionally with a suffix: 0.5, 30s, 2m, 1h.

Putting timeout around risky external commands in cron jobs and CI scripts is one of the cheapest reliability improvements you can make. A hung job that never ends is far worse than a failed one that reports an error.

Job control in an interactive terminal

In an interactive terminal (what you get when you log in or open a terminal window) Bash has job control. It numbers your jobs and lets you move them between foreground and background. These commands need a real terminal, so the following is a read-only illustration and not something our editor can run.

CONSOLE
$ sleep 300 &
[1] 4821
$ sleep 400 &
[2] 4822
$ jobs
[1]-  Running                 sleep 300 &
[2]+  Running                 sleep 400 &
$ fg %1          # bring job 1 to the foreground
sleep 300
^Z               # press Ctrl-Z: stop (suspend) it
[1]+  Stopped                 sleep 300
$ bg %1          # let it continue in the background
[1]+ sleep 300 &
$ kill %2        # signal a job by job number instead of PID
Command Meaning
cmd & Start cmd in the background; Bash prints [job] PID
jobs List the shell's jobs (-l adds PIDs, -r running only, -p PIDs only)
fg %n Bring job n to the foreground
bg %n Resume a stopped job in the background
Ctrl-Z Stop (suspend) the foreground job
Ctrl-C Send SIGINT to the foreground job
%n, %%, %+, %- Job specs: number, current job, previous job

Job control is off in scripts, which is why there are no [1] 4821 messages in our examples. In a script you work with PIDs ($!, wait, kill) rather than job numbers. The builtin jobs still lists the background children, as the throttle example above showed.

Surviving logout: nohup and disown

When you close a terminal, the kernel sends SIGHUP ("hangup") to the shell, and the shell passes it on to its jobs. By default, that kills them. Two ways to avoid that:

  • nohup cmd & starts cmd with SIGHUP ignored and, if its output would go to the terminal, redirects it to a file named nohup.out.

  • disown removes an already running job from the shell's job table, so the shell no longer forwards SIGHUP to it.

Bash nohup ignores HUP and saves the output

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The ordinary child was killed by the HUP before it could print anything; the nohup child ignored it and finished. Because we redirected the output ourselves, nohup had nothing to rewrite. Only when its standard output is still a terminal does nohup redirect it to a file called nohup.out and print nohup: ignoring input and appending output to 'nohup.out'. Redirect explicitly (nohup cmd > app.log 2>&1 &) to choose the file name.

For anything that must run for days (a web server, a queue worker) neither trick is the right tool. Use a service manager such as systemd (see Packages, Services and systemd) or run it inside a terminal multiplexer like tmux (see SSH, scp and Working Remotely). setsid cmd & is another option: it starts the command in a new session with no controlling terminal, so it never receives the terminal's SIGHUP.

Common mistakes

  • Forgetting wait. The script finishes, the shell exits, and output from background jobs may arrive late or never be checked.
  • Reading $! too late. It only holds the PID of the most recent background job. Save it right away.
  • Variables set in a background job are lost. A job runs in a copy of the shell (see Subshells), so count=$((count+1)) & changes nothing in the parent. Communicate through files or output.
  • Writing to the same file from many jobs. Lines can mix or overwrite each other. Give each job its own file and merge afterwards.
  • Ignoring status. wait with no arguments returns 0 even when jobs failed. Wait for each PID if failures matter.
  • Assuming order. Parallel jobs finish in any order; never base logic on it.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the special variable $! contain?

    $! is the PID of the last job started with &. The exit status is $?, the shell's own PID is $$.
  2. 2. You run wait with no arguments after starting three background jobs and one of them fails. What status does wait return?

    A bare wait waits for all children but returns 0 whatever they did. To learn about failures, wait for each PID separately and look at its status.
  3. 3. A command run with timeout 2 cmd is still running after two seconds. What happens, and what is the exit status?

    timeout sends SIGTERM by default and reports 124 to show that it timed out. SIGKILL is only used with --kill-after, and then the status is 137.
  4. 4. Which tool lets a script wait until any one of several background jobs finishes, so that it can limit concurrency?

    wait -n (Bash 4.3 and later) returns when the next background job ends. Together with jobs -rp it makes a simple throttle.
  5. 5. Why do fg and bg usually not help inside a script?

    Job control belongs to interactive terminals. Scripts track background work with PIDs: $!, wait and kill.

Summary

  • cmd & starts a background job; $! holds its PID, and you should save it immediately.
  • wait PID waits for one job and returns its exit status; a bare wait waits for all jobs but always returns 0.
  • Run independent work in parallel, give each job its own output file, and print results in a fixed order afterwards.
  • Throttle with jobs -rp and wait -n, or use xargs -P.
  • timeout DURATION cmd stops a command that runs too long (status 124, or 137 with --kill-after).
  • jobs, fg, bg, Ctrl-Z and %n belong to interactive terminals; nohup, disown and setsid keep jobs alive after logout.

Next: kill, Signals and Process Control