Skip to content
Tutorials Center
My learning

Topics

Lesson 98 of 134 ยท Concurrency

Processes and Subprocesses

Learn Ruby processes: fork, Process.wait, exit status, system, backticks, spawn, Open3, pipes between processes, signals and how to avoid shell injection.

A process is a running program with its own memory, its own variables and its own place in the operating system. Threads and fibers live inside a process and share its memory; processes do not share anything unless you set it up. That separation makes processes the sturdy option: a crash in one does not damage another, and each process can use a different CPU core, so they are the standard way around the GVL for CPU-bound Ruby code.

Ruby gives you two families of tools. One copies your own program (fork) so that a second Ruby process continues with the same code. The other starts other programs (system, backticks, spawn, Open3): ls, git, ffmpeg or another Ruby script. This lesson covers both, with exit codes, pipes, signals and the traps professionals watch for.

The runner here is a Linux sandbox where fork and common shell tools such as sh, echo and tr work, so every example below runs. On Windows, and on JRuby, fork is not available (Process.respond_to?(:fork) tells you). The subprocess tools work everywhere, but the commands you start are of course different on each system.

fork: a copy of the running program

fork duplicates the current process. With a block, the block runs in the child process and the child exits when the block ends; the parent gets the child's process ID (a number) back and continues. Call Process.wait(pid) to wait for the child to end, and read its result from $? (a Process::Status).

Ruby fork, wait and the exit status

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The child prints its line and exits with code 3. By convention 0 means success and anything else means failure. success? is false for code 3. Waiting is not optional: a child that has ended but whose parent never waited stays in the process table as a zombie and wastes a slot. Always wait, or use Process.detach (below) for children you do not want to wait for yourself.

Memory is copied, not shared

After fork the child has a copy of every variable. Changes in one process are invisible in the other.

Ruby The child works on its own copy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The child prints 11, the parent still has 10. There is no data race (nothing is shared) but also no easy way to return a result. You need a channel for that: an exit code (a number from 0 to 255), a file, or a pipe.

Pipes: talking between processes

IO.pipe returns two connected ends: what you write into the writer can be read from the reader. Before forking, create the pipe; afterwards, each side closes the end it does not use. That last step matters: the reader sees the end of the data only when every copy of the write end is closed.

Ruby Sending a result through a pipe

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Marshal.dump turns most Ruby objects into bytes and Marshal.load rebuilds them. Use it only between processes you trust, since loading data from an untrusted source is a security risk. For text protocols, JSON (JSON) is the safer choice.

Several workers

Starting a few children and collecting their results is the "fork pool" pattern that servers such as Puma and Unicorn grew out of. Each child sends its answer through its own pipe:

Ruby Four child processes, one result each

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The sums come back in the same order as the chunks, because we read the pipes in the order we created them. For real speed-ups you need more than one CPU core (the sandbox here has only one), and the work must be big enough to repay the cost of creating a process.

Running other programs

system: run and wait

system runs a command, waits for it, lets its output go straight to your output, and returns true (exit code 0), false (non-zero exit code) or nil (the command could not be started at all). The details are in $?.

Ruby system and its three answers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The shell reports a missing program as exit code 127, but system itself returns nil for it, so nil means "not even started". Passing exception: true turns both failures into exceptions: RuntimeError: Command failed with exit 1: exit 1 for a failing command, and Errno::ENOENT for a missing one. In scripts that must not continue after a failed step, that is far safer than forgetting to check the return value.

Backticks: capture the output

Backticks (or %x(...)) run a command and give you its standard output as a string. Standard error is not captured.

Ruby Capturing output with backticks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Unlike system, backticks raise Errno::ENOENT when the program does not exist. A non-zero exit code is silent, so check $?.success? yourself.

Avoid the shell: pass an array

When you hand system or backticks a single string, Ruby may run it through a shell (sh), and the shell interprets special characters such as ;, |, $ and *. If part of that string comes from a user, they can add their own commands. This is called shell injection, and it is one of the classic security holes.

Pass the program and its arguments as separate arguments instead. Ruby then starts the program directly, and every argument is plain data.

Ruby String versus separate arguments

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The first call prints saving report.txt and then a second line, INJECTED: the shell ran our extra command. The second call prints the whole text, saving report.txt; echo INJECTED, as one harmless argument. Make this a reflex: never build shell commands by pasting user input into a string.

Open3: output, errors and exit code together

Backticks give you only standard output. The open3 library (a default library; require "open3") captures standard output, standard error and the exit status, and can feed standard input.

Ruby Open3.capture3, capture2e and stdin_data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

capture3 returns the three results [stdout, stderr, status]; capture2 returns [stdout, status]; capture2e merges both output streams into one string, in the order they were written. stdin_data: supplies the input. For long-running programs whose output you want to process line by line, Open3.popen3 yields the three streams and a wait thread:

Ruby Open3.popen3 for streaming

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

You must close stdin for programs like cat that read until the end of their input. The block form cleans up the streams and the child process when it ends.

spawn: start now, wait later

spawn starts a command without waiting and returns its process ID. It accepts an environment hash first, then the command, then options such as out: and err: for redirecting. Pair it with Process.wait, wait2 (returns pid and status) or waitall.

Ruby spawn, wait2, waitall and detach

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Process.waitall collects every remaining child. Process.detach(pid) returns a thread that waits for you, so you never leave a zombie behind. The hash given to spawn sets environment variables for the child only. The last call, Process.wait with no children left, raises Errno::ECHILD: No child processes.

Starting another Ruby

To run Ruby as a subprocess use RbConfig.ruby, which holds the path of the interpreter currently running, instead of assuming a ruby on the PATH.

Ruby A Ruby subprocess

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Both subprocesses print their result (42 and ["alpha", "beta"]). Notice that the second call passes the arguments separately again.

Signals

A signal is a small message from the operating system or another process: INT (Ctrl+C), TERM (please stop), KILL (stop right now, cannot be caught), USR1/USR2 (free for your own use). Process.kill("TERM", pid) sends one, and trap (also called Signal.trap) installs a handler.

Ruby Sending signals to a child

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The first child catches TERM, prints, and exits with code 0. The second child cannot catch KILL, so it dies without a chance to clean up: signaled? is true, termsig is 9, and there is no exit code (exitstatus is nil). The string form reads pid N SIGKILL (signal 9). A well-behaved service first sends TERM, waits a little and uses KILL only as a last resort.

In your own process, trap("INT") replaces the default behavior of Ctrl+C. Without a trap, Ruby raises an Interrupt exception in the main thread, which you can rescue:

Ruby Receiving signals in the main process

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Keep signal handlers tiny. They run at awkward moments, and many things (locks, buffered I/O) are not safe to use inside them. A common pattern is to set a flag or push onto a queue in the handler and let the main code do the real work.

Exit codes, exit! and crashes

How a process ends determines the status its parent sees:

  • exit runs at_exit handlers and ensure blocks, and ends with code 0, or the number you give it.
  • exit! ends immediately with no handlers. After fork, use it in a child that must not run the parent's cleanup code.
  • abort("message") prints the message to standard error and exits with code 1.
  • An unhandled exception prints its message and trace and exits with code 1.
Ruby How children can end

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

You see cleanup ran only for the plain exit, and giving up printed by abort. The codes printed are 0, 42, 0, 9 and 1. Remember that exit codes are one byte (0-255): exit 256 would look like 0 to the parent.

Processes, threads or fibers?

Process Thread Fiber
Memory Separate (copied on fork) Shared Shared
Ruby code in parallel Yes, on several cores No (GVL) No
A crash affects Only that process The whole process The whole process
Communication Pipes, sockets, files, signals Variables, Queue Variables, resume
Cost Highest Medium Lowest
Typical use CPU-bound work, isolating risky tasks, running other programs Waiting on I/O Many cheap concurrent waits

On Linux, fork is cheap thanks to copy-on-write (memory pages are copied only when somebody changes them), which is why many web servers load the application once and then fork workers. Keep two risks in mind: forking a process that has running threads copies only the calling thread (locks held by other threads stay locked in the child), and open files or network connections are inherited by the child. Libraries that keep connections, such as database clients, usually need to reconnect after a fork.

For more parallelism inside one process see Parallelism and Ractors.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A child created with fork changes a variable it inherited from the parent. What does the parent see?

    Processes do not share memory. The child gets a copy, so you need a pipe, a file or the exit code to send data back.
  2. 2. What does system("no_such_program_xyz") return?

    system returns nil when the command cannot be run at all, false when it ran and failed. With exception: true a missing command raises Errno::ENOENT.
  3. 3. Why is system("convert #{params[:file]}") dangerous, and what is the fix?

    A single string can go through the shell, which allows injection of extra commands. Separate arguments are passed to the program as plain data.
  4. 4. Which Open3 method returns standard output, standard error and the exit status?

    capture3 returns [stdout, stderr, status]. capture2 omits stderr, and capture2e merges stderr into stdout.
  5. 5. Why must you call Process.wait (or Process.detach) for children you start?

    The operating system keeps the exit status of a finished child until the parent asks for it. Not asking leaves zombie entries behind.

Summary

  • fork copies the running program; the child has its own memory. Use Process.wait, wait2, waitall or Process.detach to collect children, and $? for the status (exitstatus, success?, signaled?, termsig).
  • Share results between processes with exit codes, files or IO.pipe (with Marshal or JSON), and close unused pipe ends.
  • system returns true, false or nil (could not start); backticks capture standard output and raise Errno::ENOENT for a missing program; exception: true makes system raise.
  • Pass the program and arguments as separate strings to avoid shell injection.
  • Open3.capture3 / capture2e / popen3 give you stdout, stderr, stdin and the exit status; spawn starts without waiting.
  • Signals: Process.kill, trap, Interrupt; TERM can be caught, KILL cannot.
  • Processes give true parallelism and isolation at the highest cost; fork is unavailable on Windows and JRuby.

Next: Parallelism and Ractors