Skip to content
Tutorials Center
My learning

Topics

Lesson 127 of 134 ยท Projects

Project: A Coroutine Pipeline

Build a concurrent processing pipeline in Kotlin - producer, parallel workers and aggregator connected by channels and flows, with back-pressure, cancellation and deterministic, testable results.

Many programs are really assembly lines: one stage reads or generates items, the next stage transforms them, a third stage collects the results. When the middle stage is slow (it calls a service or does heavy work), you want several workers at once. When one stage fails, you want everything to stop cleanly. Coroutines, channels and flows are built for exactly this.

In this project you build such a pipeline in four steps and finish with a log-processing program. It brings together coroutine builders, structured concurrency, cancellation, channels, flows and shared state. You should have read those lessons first. It follows the text adventure and is the last project before the JSON parser.

The idea

TEXT
 producer  --channel-->  worker 1 --\
                         worker 2 ---+--channel-->  aggregator  --> report
                         worker 3 --/

Each arrow is a channel (or a flow). A stage never calls another stage directly; it only sends to and receives from a pipe. This has three nice effects:

  • Decoupling. A stage does not know who is on the other side.
  • Back-pressure. If a channel is full, the sender suspends until there is room. A fast producer cannot flood memory.
  • Parallelism where you want it. Several workers can read from the same channel, and every item goes to exactly one of them.

The runner has one CPU and no real network, so we simulate slow work with delay. delay suspends a coroutine without blocking a thread, so the pipeline behaves like a real one that waits for I/O. The printed results are always sorted or aggregated so that they do not depend on scheduling.

Step 1: two stages connected by channels

produce starts a coroutine that returns a ReceiveChannel. The channel is closed automatically when the coroutine's block ends, so the consumer's for loop stops by itself. We write each stage as an extension function on CoroutineScope, so the stage becomes a child of whatever scope starts it (structured concurrency).

Kotlin Step 1 - producer and transformer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with readln()), it gets the next line from this box.

Notice what is missing: no close() calls, no thread, no lock. When numbers runs out of numbers, its channel closes, so the for loop in squares ends, so that channel closes too, so the loop in main ends. Closing flows along the pipeline like a domino chain.

Step 2: fan-out with parallel workers

Now we make the middle stage slow and run several copies of it. All workers read from the same jobs channel; Kotlin guarantees that each item is received by exactly one worker. This is called fan-out. The results go to a second channel, which is fan-in.

The important detail is when to close the results channel: only after all workers are finished. A coroutineScope waits for its children, so closing right after it is correct.

Kotlin Step 2 - fan-out and fan-in

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with readln()), it gets the next line from this box.

Which worker handled which job is up to the scheduler, and it changes from run to run, so we do not print it. What is deterministic is the outcome: all 12 jobs, each exactly once, which we make visible by sorting. A good concurrent program has a deterministic result even though its schedule is not. If your output changes from run to run, you probably have a race (see shared mutable state).

Two design points:

  1. The results channel is UNLIMITED here because we only read it after the workers finish. With a rendezvous channel the workers would suspend forever on send, and nobody would be reading, which is a deadlock. Always check who drains each channel and when.
  2. The jobs channel is a rendezvous channel (capacity 0): a worker takes a job only when it is free. That is back-pressure at work.

A mistake: closing too early

Imagine you wrote results.close() at the end of each worker. The first worker to finish would close the channel, and the other workers' send would then throw ClosedSendChannelException. Close from the one place that knows all senders are done: the code that waits for them.

Step 3: a flow version with limited parallelism

Channels give you full control. Flows give you a more declarative style: a cold stream with operators. For "process N items with at most K at a time" a channelFlow plus a Semaphore is compact and uses only stable APIs.

A semaphore lets only K coroutines into a section at once. Unlike a fixed set of workers, it starts one coroutine per item, and the semaphore limits how many run at the same time.

Kotlin Step 3 - limited parallelism with a flow

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with readln()), it gets the next line from this box.

The function enrichAll hides all the concurrency. Callers get a Flow<Enriched> and do not care about semaphores. Because the flow is cold, nothing starts until someone collects it, and when the collector stops (for example with take(5)), the producer coroutines are cancelled automatically.

Choosing between the two shapes:

Use When
Channels with a worker pool You want a fixed number of long-lived workers, explicit queues and explicit closing.
channelFlow + semaphore, or flatMapMerge Items are independent, you want operators (map, filter, take) and cancellation from the collector.
Plain map { async { ... } }.awaitAll() A small, bounded list. No streaming needed.

Step 4: failure and cancellation

In a pipeline, one failing stage should stop the others. Structured concurrency does that for you: an uncaught exception in a child cancels the parent scope, which cancels all siblings. The siblings get a CancellationException at their next suspension point, and their finally blocks run. We use that to release resources.

Kotlin Step 4 - one stage fails, all stop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with readln()), it gets the next line from this box.

The consumer fails on record 4. The scope cancels the producer, which was in the middle of its loop. Both finally blocks run before the exception reaches the catch outside. The producer never sends most of its hundred items. Without structured concurrency (for example with GlobalScope.launch), the producer would keep running forever and leak.

Notice where the try/catch is: around coroutineScope, not inside the launch. Exceptions inside launch do not travel through try/catch around the launch call. See exceptions in coroutines.

For "one bad item should not stop the others", use a different strategy: catch the exception inside the worker and send a result value that says it failed (a sealed type), as we do in the final program. Use supervisorScope only when siblings are truly independent.

A time limit for the whole pipeline

Pipelines that wait on the outside world need a deadline. withTimeout cancels the block when time is up and throws TimeoutCancellationException; withTimeoutOrNull returns null instead.

Kotlin
// read-only example
val report = withTimeoutOrNull(2_000) {
    runPipeline()          // a suspend function
} ?: run {
    println("Pipeline took too long and was cancelled")
    emptyReport()
}

Step 5: a compile error you will see

Suspension is part of a function's type. A function that calls delay must itself be suspend. This program fails on purpose.

Kotlin delay in a normal function (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with readln()), it gets the next line from this box.

The compiler says that the suspend function delay can only be called from a coroutine or another suspend function. Mark fetchPrice as suspend and the code compiles. This "colored function" rule is a feature: the signature tells every reader that the call can pause.

The finished program: a log analyzer

We finish with a complete pipeline. A producer creates raw log lines. Three parse workers turn them into ParseResult, which is either a parsed entry or a failure (so one bad line never kills the pipeline). A single aggregator coroutine owns the statistics map. Because only one coroutine ever touches the map, we need no lock: this is confinement, the safest form of shared state.

Kotlin The log pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with readln()), it gets the next line from this box.

Let us walk through the design decisions, because they are the lesson:

  • Bounded channels (capacity = 8) keep memory use constant even for a billion lines. The producer is suspended whenever the workers are behind.
  • The inner coroutineScope inside the second launch waits for the three workers. Only then is parsed closed. The aggregator's for loop ends exactly when that happens.
  • The aggregator runs in the function's own coroutine, so runPipeline returns only when everything is complete. No join is needed.
  • Failures are data. ParseResult.Failed flows through the same channel as successes. A bad line produces a line in the report, not a crashed pipeline.
  • Confinement instead of locks. The Stats objects are mutable (var), yet it is safe, because only the aggregator coroutine touches them. The report is built after the loop.
  • A deadline: withTimeout(5_000) wraps the whole run, so a stuck stage cannot hang the program.
  • Determinism. The map is sorted by service name, failures are sorted by line, and the counts do not depend on the order of arrival. Only the time and the interleaving are non-deterministic, and we print neither.

Do not add println in the workers to "see what is happening" and then compare output between runs. The interleaving of lines from concurrent coroutines is not stable. Log with a coroutine name (CoroutineName) when you debug, and assert only on aggregated results in tests.

Performance notes

  • On a real multi-core machine, CPU-heavy workers need Dispatchers.Default (about one thread per core). For blocking calls such as JDBC, use Dispatchers.IO. The pipeline code stays the same (dispatchers and context).
  • More workers help only while the stage waits for something. For pure computation, workers beyond the number of cores add overhead.
  • A channel hop costs a few hundred nanoseconds. If each item takes microseconds, batch them (chunked) so a worker processes 100 items per message.
  • Measure with a realistic workload before tuning buffer sizes. A buffer of 8 to 64 is usually enough.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What happens when several workers loop over the same Channel with for (job in jobs)?

    A channel hands each element to a single receiver. Several receivers share the work (fan-out).
  2. 2. When should the shared results channel be closed in the fan-out example?

    Closing from a single place that knows all senders are done avoids ClosedSendChannelException in the remaining workers.
  3. 3. Why does a bounded channel (capacity = 8) help in a pipeline?

    send suspends when the buffer is full. That is back-pressure.
  4. 4. Why is it safe to mutate the Stats objects in the log pipeline without a lock?

    Confinement to one coroutine is the simplest form of safe shared state.
  5. 5. A stage throws an exception inside coroutineScope. What happens to its sibling coroutines?

    Structured concurrency cancels siblings when a child fails. Cleanup in finally still runs.

Summary

  • A pipeline is stages connected by channels or flows; closing the first stage lets the end of the data flow through the whole chain.
  • Fan-out means several workers read one channel; fan-in means several senders write one channel. Close the shared channel only after all senders are done.
  • Bounded channels give back-pressure and constant memory.
  • channelFlow with a Semaphore limits parallelism declaratively, and cancels cleanly when the collector stops.
  • Structured concurrency stops the whole pipeline when a stage fails; model expected per-item failures as data instead.
  • Keep mutable state confined to one coroutine, make the result deterministic, and wrap the pipeline in withTimeout.

Next: Project: A JSON Parser