Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 66 ยท Threads and Concurrency

Thread Pools and Implicit Threading

Learn why servers use thread pools instead of one thread per request, how work queues, fork-join and thread-local storage work, and compare process-based and thread-based multitasking.

Creating a thread per task sounds simple, but at scale it fails. A web server that spawns a new thread for each of 10,000 incoming requests pays the creation cost every time, can run out of memory (each thread needs a stack), and wastes time switching between thousands of threads. The standard solution is the thread pool: create a fixed number of worker threads once, keep them alive, and feed them tasks through a queue.

This lesson covers thread pools and the related ideas known as implicit threading, where a library or runtime manages threads for you: fork-join, parallel loops and thread-local storage. At the end it compares process-based and thread-based multitasking, which is a common exam question.

The thread pool idea

Task queue Requests wait in line.
Worker threads A fixed set takes tasks one by one.
Results Returned or stored when done.

Threads are created once and reused.

1 / 2

How a thread pool works

flowchart LR
    Client["Incoming tasks"] --> Q["Task queue"]
    Q --> W1["Worker 1"]
    Q --> W2["Worker 2"]
    Q --> W3["Worker 3"]
    W1 --> R["Results"]
    W2 --> R
    W3 --> R

Idle workers pull the next task from the queue; if all are busy, tasks wait.

  1. At start-up the pool creates N worker threads that all wait on a shared queue.
  2. A client submits a task (a function plus arguments). It goes into the queue.
  3. An idle worker takes it, runs it, stores the result, and returns to waiting.
  4. If more tasks arrive than workers exist, the extras queue up instead of creating more threads. Some pools can grow up to a maximum and shrink again when idle.

Choosing the pool size

  • CPU-bound tasks: about the number of cores (more only adds switching).
  • I/O-bound tasks: more than the number of cores, because most threads are waiting. A common estimate is cores ร— (1 + wait time / compute time).

A thread pool simulation

Real threads are not available in this browser sandbox, so the pool below is a simulation on a virtual clock. It processes 8 tasks with three workers and shows which worker handled each task and when, including the time tasks spent queued.

Python Simulating a thread pool

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On a real system you would use ready-made pools: Java's ExecutorService, Python's concurrent.futures.ThreadPoolExecutor, .NET's ThreadPool, Windows' thread pool API, or Apple's Grand Central Dispatch. For example, on a regular Python install:

Python
from concurrent.futures import ThreadPoolExecutor

def fetch(n):
    return n * n

with ThreadPoolExecutor(max_workers=3) as pool:
    print(list(pool.map(fetch, range(8))))

Fork-join parallelism

In the fork-join model, a task splits itself into subtasks (fork), runs them in parallel, then waits for them and combines the results (join). It fits divide-and-conquer algorithms such as merge sort. Java's ForkJoinPool adds work stealing: an idle worker steals subtasks from a busy worker's queue, balancing load automatically.

The next demo sums a list by recursive splitting, the same structure a fork-join framework would run in parallel.

Python Fork-join in miniature

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other forms of implicit threading

  • Parallel loops and streams: OpenMP's #pragma omp parallel for, Java parallel streams and .NET Parallel.For split loop iterations across threads for you.
  • Task-based and async models: you describe what to run; the runtime decides how (coroutines, async/await, Grand Central Dispatch blocks).
  • Thread-local storage (TLS): each thread gets its own copy of a variable, so shared-looking names do not need locks. Typical uses: per-thread error codes (errno), random-number state, request context in a server.
Python Thread-local data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Process-based vs. thread-based multitasking

Both let a computer appear to do many things at once. They differ in granularity.

Aspect Process-based multitasking Thread-based multitasking
Unit Whole program (a process) A part of one program (a thread)
Example Listening to music while editing a document One editor checking spelling while you type
Memory Separate address spaces Shared address space
Overhead Higher (heavy switching) Lower
Communication IPC Shared variables
Failure One process crashes alone One thread can take down the process

Real systems combine both: many processes, each with several threads.

When not to use threads

  • For heavily CPU-bound work in languages with a global lock (Python's GIL in the default build, for instance), use processes or native extensions instead of threads.
  • When the task is trivial, a queue and a thread's overhead can cost more than the task.
  • When single-threaded event loops (Node.js, asyncio) already give enough concurrency for I/O.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main advantage of a thread pool over creating a new thread per task?

    Workers are created once and reused; the pool size bounds the load.
  2. 2. A pool has 4 workers and 10 tasks arrive at once. What happens to the other 6 tasks?

    Tasks wait in the work queue; this is how the pool limits concurrency.
  3. 3. What does work stealing do in a fork-join pool?

    It balances the load automatically.
  4. 4. Which statement about thread-local storage is correct?

    TLS gives each thread private data under a shared name.
  5. 5. Which is an advantage of thread-based multitasking over process-based?

    Threads share one address space, so switching and communication are cheap, at the cost of weaker isolation.

Summary

  • A thread pool reuses a fixed set of workers fed by a task queue, avoiding per-task thread creation and runaway thread counts.
  • Pool size depends on whether work is CPU-bound (about the core count) or I/O-bound (more).
  • Fork-join, parallel loops and async tasks are forms of implicit threading; work stealing balances load.
  • Thread-local storage gives each thread private data under a common name.
  • Process-based multitasking is heavier and safer; thread-based is lighter and riskier; real systems mix both.

Next: CPU Scheduling Basics