Skip to content
Tutorials Center
My learning

Topics

Lesson 122 of 132 ยท Projects

Project: A Concurrent Job Pipeline

Build a concurrent crawler in Rust with worker threads and channels: share a job queue, aggregate results, apply backpressure, retry failures, survive panics and shut down cleanly.

Most useful concurrent programs have the same shape: some work arrives, a group of workers processes it in parallel, and the results flow back to one place that combines them. A web crawler is the classic example. It fetches a page, finds the links on it, and sends every new link back into the queue, until nothing new turns up. Search engines, link checkers, build tools and data importers all follow this pattern.

In this project we build such a crawler with plain threads and channels from the standard library. To keep it runnable (and deterministic) in the lesson editor, the "web" is a small in-memory site, with a missing page, a page that times out the first time, and a page whose parser has a bug. The concurrency problems are the real ones, though:

  • How do workers share one queue of jobs?
  • How does the coordinator know the crawl is finished when work keeps spawning more work?
  • How do you stop cleanly, without losing results or leaving threads behind?
  • How do you keep a slow or crashing job from taking everything down?
  • How do you make sure that concurrency did not change the answer?

The last question is the most important one, and we answer it with a test: the parallel crawler must produce exactly the same report as a simple single-threaded one.

Output order from concurrent code is only guaranteed where the code enforces it (joining, sorting, sequence numbers). Every example below prints results in an order that is fixed by the program, never by thread scheduling. Where something does vary between runs, the text says so.

Stage 1: the site and a sequential baseline

First the data. Each page has a URL, some text and a list of outgoing links. The pages form a graph with cycles (/guide links back to /), a dead link (/missing) and a flaky page that fails the first time it is requested.

Fetching a page can fail in two ways, so it returns a Result. A Timeout is worth retrying, a NotFound is not. Parsing a page counts its words and extracts its links. Then the baseline: a breadth-first crawl with a queue and a set of visited URLs, which is exactly what you would write without threads.

Rust Stage 1: the site, fetch, parse and a sequential crawl

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

Three details to notice before we add threads:

  • visited.insert(link.clone()) returns true only the first time a URL is seen. It does the "have we been here?" check and the marking in one step, and it is what keeps the crawl from looping forever on the cycles.
  • The Report is built only from BTreeMaps. They iterate in sorted order, so the report prints the same way every time, and two reports can be compared with ==. That equality is how we test the concurrent version.
  • The pages are static data. Threads may read 'static data freely, with no Arc and no lock, because nobody can mutate or free it.

Stage 2: workers, channels and knowing when to stop

Now the parallel version. The design has three moving parts:

  1. A job channel. The coordinator sends Job { url, attempt }. All workers receive from the same channel. A Receiver cannot be cloned, so it is wrapped in Arc<Mutex<Receiver<Job>>>, and a worker locks it only for the moment it takes the next job.
  2. A result channel. Workers send an Outcome back to the coordinator. Many workers, one receiver: Sender clones are cheap and designed for this.
  3. A coordinator (the calling thread) that owns the visited set and the report. It is the only one that touches them, so they need no lock at all. Workers do the slow, independent part (fetch and parse); one thread does the bookkeeping. This "share by communicating" design is much easier to get right than letting every worker lock a shared set.

The hard question is termination. A job can produce more jobs, so "the job queue is empty" does not mean "we are done": a worker may be processing a page that is about to announce ten new links. The reliable rule is to count jobs in flight: increase the count when you send a job, decrease it when its outcome arrives. When it reaches zero, nothing is queued, nothing is running, and nothing can produce new work.

Retries are driven by the coordinator in this version: a Timeout outcome makes it send the same job again with attempt + 1.

Rust Stage 2: a worker pool and a coordinator with in-flight counting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

Now trace what happens, because this is the heart of the project:

  • Start: one job (/) is sent and in_flight is 1. A worker takes it, fetches and parses, and sends an Outcome containing the links.
  • Each outcome decrements in_flight and then maybe increments it again once per new link or retry. The counter can only reach 0 when the last outcome arrives and it spawned nothing.
  • Order of arrival is arbitrary. With 8 workers, outcomes come back in different orders on different runs. That is fine, because the report is built from order-independent operations: inserting into maps and adding counts. This is the property that makes the parallel result equal the sequential one.
  • Shutdown: drop(job_tx) closes the job channel. Each worker's recv() then returns Err, the worker breaks out of its loop and returns, and join waits for it. Nothing is leaked, and no thread is killed.
  • The coordinator drops its own outcome_tx. If it kept one, a crash of all workers would leave recv() waiting forever for a message that no one can send. With the sender dropped, recv() returns an error instead, and the expect turns a hang into a clear panic.

The most common bug in this design is forgetting a sender. A channel is closed only when every Sender clone is dropped. If the original outcome_tx stays alive in the coordinator, a loop such as for outcome in outcome_rx never ends. When a program that uses channels hangs at the end, look for the sender that is still alive.

Stage 3: pipelines, backpressure and cancellation

The crawler has a feedback loop (results create new jobs). Many programs are simpler: data flows one way through stages, each stage with its own threads, connected by channels. For example, read pages โ†’ tokenize โ†’ count. Two ideas from this shape matter a lot:

  • Shutdown by closing. A stage's loop is for item in rx { ... }, which ends by itself when the previous stage finishes and drops its sender. A single drop at the beginning of the pipeline cascades down to the end, and every thread exits in order. No flags, no special "poison" messages.
  • Backpressure. An ordinary mpsc::channel is unbounded: a fast producer can fill memory while a slow consumer lags. A mpsc::sync_channel(n) holds at most n items, and send blocks when it is full. The slow stage thereby slows the fast one down, which is exactly what you want.

Parallel stages finish in arbitrary order, so the last stage restores the original order using a sequence number and a small buffer.

Rust Stage 3a: a three-stage pipeline with bounded channels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

The documents are tokenized by whichever worker is free, so index 5 may be finished before index 2. The buffer waiting holds early arrivals until next shows up, so the lines print in order 0, 1, 2 and so on, every time. Notice how the program ends by itself: the producer finishes and drops doc_tx, the tokenizers see the closed channel and exit, dropping their tok_tx clones, and finally for ... in tok_rx ends. We never told any thread to stop.

Closing channels handles "no more work". A different problem is "stop now, even though there is more work": a user presses Ctrl-C, or a deadline passes. For that, use cooperative cancellation: a shared flag that workers check between small units of work. Threads in Rust cannot be killed from outside (and they should not be), so they must agree to leave.

Rust Stage 3b: cooperative cancellation with an atomic flag

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

The program receives exactly four start messages instead of looping over the receiver until it closes: the workers still hold their senders while they run, so the channel would not close until they stopped, which they only do after the flag is set. Ordering::Relaxed is enough for a plain flag whose only job is to say "stop": no other data depends on seeing the flag at a particular moment. If the flag also published data (for example "the result is ready"), you would use Release on the store and Acquire on the load. See atomics. The crucial design rule is the granularity of the check: a worker that checks the flag once per page reacts within one page; a worker that checks once per hour does not.

Stage 4: the complete crawler

The last version adds the things that production pipelines need:

  • A depth limit. The crawl proceeds in waves: all pages at depth 0, then all pages at depth 1, and so on, up to max_depth. Within a wave the workers run in parallel, but between waves the coordinator waits. A wave-based crawl is less aggressive than the free-flowing one from stage 2, and it has a big advantage: its result is deterministic even with a depth limit, so we can compare it with the sequential crawler.
  • Retries inside the worker. Simpler than the coordinator-driven version: the worker retries a timed-out page itself, up to three attempts.
  • Panic isolation. The page /crash triggers a bug in our "parser". Each job runs inside catch_unwind, so the panic becomes a FetchError::Panicked value for that page, and the worker survives.
  • Statistics collected with atomic counters shared by all workers.
Rust Stage 4: the complete concurrent crawler

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

Check the output against the code:

  • Every row says same as sequential = true. The parallel crawler is an optimization, not a different program: with the same input it must produce the same answer. If you ever see false, you have a race or an order dependence somewhere.
  • Jobs equals the number of distinct URLs fetched, and retries is 1 whenever /flaky was reached (its first attempt times out, the second succeeds). Both numbers are the same on every run, because each URL is fetched exactly once, even though the order differs.
  • Waves tells how many levels were crawled. With max_depth large, the number stops growing when the frontier runs out.
  • The failures are /missing (NotFound) and /crash (Panicked). The panic did not stop any worker, and join never returned an error.
  • Statistics use atomic counters because many workers update them at the same time. They are read only after the workers have been joined, so even the cheapest ordering would be correct. SeqCst is a safe default when you are not sure.

What makes this design work

  1. One owner for the shared state. The visited set and the report live in the coordinator only. Workers get a URL, return a value, and never touch shared collections. No Mutex guards any of that.
  2. Messages, not shared memory. The only shared structures are the job queue (a receiver behind a mutex, locked for an instant) and a few atomics.
  3. Termination by counting. Stage 2 counts jobs in flight; stage 4 counts exactly how many results to expect in a wave. Never use "the queue looks empty" as a termination test: a worker may be about to add work.
  4. Shutdown by dropping. Closing the sender is the universal "no more work" message. Joining the handles then guarantees that nothing is still running.
  5. Failures are values. A missing page, a timeout and a panic all become entries in the report, instead of crashing the crawl.
  6. Determinism by construction. Sorted maps, sorted waves and a comparison with a sequential reference make the program testable, even though its execution is not repeatable.

Mistakes to avoid

  • Holding the receiver lock while working. while let Ok(job) = rx.lock().unwrap().recv() { work(job) } keeps the guard alive through the loop body, which serializes all workers. Take the message in a separate let.
  • Unbounded queues. With a fast producer and slow consumers, an unbounded channel is a memory leak waiting to happen. Use sync_channel when producers can outrun consumers.
  • Deadlock through bounded channels. If a stage sends results back into a bounded channel that it also reads from, and both are full, everything waits for everything. The crawler avoids it because the job channel is unbounded and the coordinator is the only sender into it.
  • Spawning one thread per job. A thousand pages do not need a thousand threads. A fixed pool is predictable. (Scoped threads, from parallel patterns, are the best fit for a fixed, known set of tasks.)
  • Ignoring join results. A worker that panicked reports it only through join(). Check it, or isolate panics as we did.
  • Printing from workers. Output from different threads interleaves unpredictably. Send results to one thread and print there.

Where to go from here

  1. Make fetching slower with thread::sleep in fetch and measure (with Instant, printing only the relative speedup) how the number of workers changes the total time. Why does it stop improving? (Hint: the machine in the lesson editor has one CPU, and the work is sleeping, not computing.)
  2. Add a politeness limit: at most two requests per host at a time. A Mutex<HashMap<Host, usize>> or a semaphore built from a Condvar would do.
  3. Replace the per-worker retries with exponential backoff.
  4. Use thread::scope to run the workers so they can borrow the report directly, with no Arc.
  5. Rewrite the coordinator with async tasks and see what changes. The structure (queue, workers, counting) stays the same; see async patterns.

The building blocks used here are explained in threads, channels, Mutex and RwLock, atomics, and Send and Sync.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does the stage 2 coordinator count jobs in flight instead of stopping when the job queue is empty?

    A worker that is processing a page may be about to announce new links. The crawl is finished only when nothing is queued and nothing is running, which is exactly when the in-flight counter reaches zero.
  2. 2. A program that uses for item in rx never ends, although all workers have finished. What is the most likely cause?

    A channel closes only when every Sender is dropped. A forgotten clone keeps it open and the receiving loop waits forever.
  3. 3. What does mpsc::sync_channel(2) give you compared to mpsc::channel()?

    A bounded channel makes a fast producer wait for a slow consumer, which keeps memory use under control.
  4. 4. How does the stage 4 crawler keep one buggy page from crashing the whole crawl?

    catch_unwind converts the unwinding panic into a value at the job boundary. The worker thread survives and the failure ends up in the report.
  5. 5. Why does the test compare the parallel report with the sequential one?

    The sequential crawler is a simple reference. If the parallel version ever differs, there is a race or an order dependence, which the comparison exposes.

Summary

  • A concurrent job pipeline has a job queue, a pool of workers, a result channel and one coordinator that owns the shared state, so most of that state needs no lock.
  • Share a receiver between workers with Arc<Mutex<Receiver<T>>>, and take each message in its own statement so the lock is released before the work starts.
  • When work creates work, count jobs in flight (or the exact number of expected results per wave) to know when to stop.
  • Shut down by dropping the sender and joining the workers; remember that a channel closes only when all Sender clones are gone.
  • Use sync_channel for backpressure, sequence numbers to restore order, and an atomic flag for cooperative cancellation.
  • Isolate failures with catch_unwind and make them values in the result.
  • Build results from order-independent operations (sorted maps, sums) and test them against a sequential reference.

Next: Rust Cheat Sheet