Skip to content
Tutorials Center
My learning

Topics

Lesson 96 of 134 ยท Concurrency

Thread Pools and Parallel Work

Build a Ruby thread pool with Queue, run jobs in parallel with ordered results, handle errors in worker threads, limit concurrency and use Timeout safely.

Starting one thread per job is fine for three jobs and a bad idea for three thousand. Every thread costs memory, the operating system has to schedule them all, and an API you call in parallel may block you if you send 3,000 requests at once. The standard answer is a thread pool: a fixed number of worker threads that take jobs from a shared queue, one after the other, until the queue is empty.

This lesson builds a small pool from the pieces you already know (Threads, Mutex, Queue and ConditionVariable), turns it into a parallel map that returns results in order, shows how errors travel out of worker threads, limits concurrency with a semaphore, and uses Timeout to stop work that takes too long, including why Timeout has sharp edges.

Everything here relies on threads overlapping their waiting (network, disk, sleep, subprocesses). Because of the GVL, a pool does not make pure-Ruby number crunching faster. For that, see Processes and Subprocesses and Parallelism and Ractors. The examples use sleep to stand in for slow I/O.

The idea in one picture

Producers put jobs into a queue. N workers each loop: take a job, run it, take the next. When the queue is closed and empty, the workers leave their loop and the threads end. The pool size, not the number of jobs, decides how much runs at once.

A minimal pool

A job here is just a block (a Proc) that the worker calls. submit puts one into the queue; shutdown closes the queue and waits for the workers to finish everything.

Ruby A small thread pool

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Three workers handle eight jobs. Each job writes into its own slot of the squares array (index i), so no two threads touch the same element and no lock is needed. The array has to be created before the jobs run; otherwise the threads would race to grow it. Which worker took which job is up to the scheduler, but the finished array is always the same.

shutdown closes the queue so pop returns nil once the backlog is empty, and the while loop ends. That is exactly the close pattern from the last lesson.

A parallel map with ordered results

Most of the time you do not need a pool class. You have a list of items, you want a result for each one, and you want them in the same order. A small helper does it:

Ruby pmap: a parallel map with a worker limit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Each job remembers its index, so the result lands in the right place, whatever the finishing order. Eight jobs of 0.1 seconds with four workers take about two rounds, around 0.2 seconds, instead of 0.8. We print comparisons rather than raw durations because timings vary between machines.

Two details to copy:

  • The helper uses yield inside the thread's block. That works: the block passed to pmap is captured and still callable from the worker threads, because blocks are closures.
  • It keeps the order of results and never prints from worker threads. Output from several threads interleaves unpredictably; return data and print it from one place.

How many workers?

  • Waiting-heavy jobs (HTTP calls, database queries, file transfers): more workers than CPU cores is fine. 5 to 50 is common, limited by what the remote side tolerates, not by your CPU.
  • CPU-heavy jobs in pure Ruby: a pool will not help because of the GVL. Use processes.
  • Unknown mix: start small, measure, and make the pool size a setting you can change.

More threads is not always faster. Past a point, extra threads only add scheduling work and memory use.

Errors in worker threads

A job can fail. What should the pool do? The wrong answer is "let the worker thread die": the pool silently shrinks, jobs wait in the queue, and the program hangs or loses work. Think about each failure in advance.

Option 1: one thread per job, collect with value. If each job is its own thread, value re-raises its error exactly where you collect the results. You decide there whether one failure should stop everything.

Ruby value re-raises a job's error (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The first job prints 50; collecting the second one raises its stored exception in the main thread and the program ends with main.rb:4:in 'Integer#/': divided by 0 (ZeroDivisionError), so the third result is never printed. That may be what you want or not. When you want all the results, wrap value in a rescue:

Ruby Collecting every result, good or bad

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Option 2: a pool, so catch inside the worker. In a pool, a job that raises would kill its worker. Rescue inside the loop and store the failure as data next to the successes:

Ruby A pmap that records errors instead of dying

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Every job produces an answer, either [:ok, value] or [:error, message], and the workers keep going. The caller can now retry, skip or report the failures. The rescue clause is narrow (StandardError), so interrupts and exit still work.

A worker that dies from an uncaught exception takes its share of the pool with it. If every worker dies, jobs stay in the queue forever, and shutdown may wait for a result that never comes. Rescue inside the job loop, and decide deliberately what failure means: retry, record, or stop the pool.

Limiting concurrency without a pool

Sometimes one thread per job is simplest, but you must not run more than N at a time (for example, at most 2 requests to a fragile server). A SizedQueue works as a semaphore: push a token before starting, pop it when done. When the queue is full, the next thread waits.

Ruby At most two jobs at once

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Six threads were created, but at most two ever ran the body at the same time: the peak is 2. The ensure returns the token even if the body fails, which is the same discipline as synchronize. This style is handy for a handful of jobs; for thousands, a pool with a few real workers is cheaper than thousands of mostly sleeping threads.

Timeout: giving up on slow work

The timeout library (a default library; require "timeout") runs a block and raises an error if it takes too long. The error is Timeout::Error with the message execution expired.

Ruby Timeout.timeout (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The first call returns the block's value (fast). The second ends the program with timeout.rb:40:in 'Timeout::Error.handle_timeout': execution expired (Timeout::Error) and a trace that ends in your own main.rb:4. Below it Ruby prints the original cause, main.rb:4:in 'Kernel#sleep': execution expired (Timeout::ExitException), which shows the line that was running when time ran out. You can pass your own exception class and message as second and third arguments, and nil or 0 means "no limit".

Ruby Handling timeouts, custom errors and cleanup

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

ensure blocks run, so files and locks inside the block are released. A timeout can be used inside a worker thread so that one stuck job does not block the whole pool.

Sharp edges of Timeout

Timeout works by raising an error inside the running code from another thread, at an arbitrary point. That has consequences you should know:

  • It can interrupt anything, even in the middle of updating shared data. Cleanup must be in ensure, and the data may be half-changed.
  • It cannot stop code that never lets Ruby run. A long-running C extension call or a blocking system call outside Ruby's control can ignore it for a long time.
  • Rescue clauses inside the block matter. On the Ruby used here a bare rescue => e inside the block did not catch the timeout, but that is an implementation detail, so keep rescue clauses inside timed code narrow and never rescue Exception there.
  • It is only a last resort. Where the library offers its own timeout (Net::HTTP has open_timeout and read_timeout, sockets and queues have timeouts), use that: it stops cleanly at a safe point. Queues even take one directly: queue.pop(timeout: 2).

Putting it together: a worker pool with results and errors

A common shape: run many independent tasks, limit parallelism, bound each task in time, and gather one outcome per task.

Ruby Bounded, timed, ordered

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The slow task is cut off after 0.1 seconds instead of five, the invalid number becomes an :error entry with Ruby's real message, and the output follows the order of the task list. This is about thirty lines and covers many real needs.

Alternatives and when to use them

  • Thread + value per job for small, bounded job lists.
  • Your own pool (as above) when the job list is long, or when jobs arrive over time.
  • The concurrent-ruby gem (not available in the runner; add it to your Gemfile in a real project) offers a battle-tested ThreadPoolExecutor, futures and promises. Use it in production code instead of maintaining your own.
  • Fibers with a scheduler when you have thousands of simultaneous waits (Fibers and the Fiber Scheduler).
  • Processes or Ractors when the work is CPU-bound.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why use a pool of workers instead of one thread per job for 10,000 jobs?

    A pool caps concurrency and reuses threads. It does not give parallel speed-up for pure Ruby code because of the GVL.
  2. 2. In the pmap helper, how do results stay in the original order although workers finish in any order?

    Writing to distinct indices of an array that was created beforehand does not conflict, and the index preserves the order.
  3. 3. A job in a worker pool raises an uncaught StandardError. What is the danger?

    An exception that escapes the loop ends the thread. Rescue inside the loop and store the failure as a result.
  4. 4. What does Timeout.timeout(1) { work } raise when the block takes too long?

    The block is interrupted, ensure blocks run, and Timeout::Error (message: execution expired) is raised. You can supply your own class and message.
  5. 5. How can a SizedQueue limit the number of jobs running at once to 2?

    push waits when the queue is full, so no more than two tokens (and therefore jobs) exist at the same time. Pop the token in ensure.

Summary

  • A thread pool is N worker threads looping on jobs.pop with a shared Queue; close plus join shuts it down cleanly.
  • For a parallel map, pair each item with its index and write each result into its own slot of a pre-sized array to keep the order.
  • Pools help with waiting (I/O), not with CPU-bound Ruby code; choose the size by what the remote side tolerates.
  • Errors: Thread#value re-raises a job's exception; inside a pool, rescue in the loop and store [:ok, ...] or [:error, ...], or the worker dies.
  • A SizedQueue of tokens acts as a semaphore to limit concurrency.
  • Timeout.timeout(seconds) { ... } raises Timeout::Error (execution expired) and runs ensure blocks, but interrupts at arbitrary points, so prefer library-specific timeouts where they exist.

Next: Fibers and the Fiber Scheduler