Skip to content
Tutorials Center
My learning

Topics

Lesson 55 of 61 ยท Projects

Project: Async Event Pipeline

Build a stream pipeline in Dart: an async* source, where and asyncMap stages, batching, back-pressure with await for, and graceful or hard cancellation.

Real programs are full of event flows: sensor readings, log lines, clicks, messages from a server. You rarely want to handle each event the moment it arrives and forget it. You want to filter it, enrich it (often with an asynchronous lookup), group events into batches, and write the result somewhere slow, all without running out of memory when the writer cannot keep up, and with a clean way to stop.

That is a pipeline, and Dart's streams are built for it. In this project you build one in stages and answer three practical questions: how do stages connect, what stops a fast source from flooding a slow sink (back-pressure), and how do you shut everything down properly (cancellation).

The Dart code is read-only because Dart is not installed on the site. TypeScript's async generators (async function* and for await) work almost exactly like Dart's async* and await for, so the stand-ins run for real and behave like their Dart twins.

The shape of the pipeline

flowchart LR
    S["Source<br/>sensorEvents()"] --> W["where<br/>filter"]
    W --> A["asyncMap<br/>calibrate"]
    A --> B["batch(3)"]
    B --> K["Sink<br/>slow writer"]
    K -. "pause / resume" .-> B
    B -. "pause / resume" .-> A
    A -. "pause / resume" .-> W
    W -. "pause / resume" .-> S

Events flow right; demand (pause and resume) flows left. A slow sink slows the whole chain.

Each box is a function from one stream to another. That is the key design idea: a stage takes a Stream<A> and returns a Stream<B>, so stages snap together like pipes. The events themselves are records, so no class is needed:

DART
typedef Event = ({int id, String sensor, double value});

Stage 1: a source with async*

An async* function is a stream generator. Each yield produces an event, and the function body is suspended there until the listener is ready for more. That suspension is the secret behind back-pressure.

DART
Stream<Event> sensorEvents({int count = 12}) async* {
  try {
    for (var id = 1; id <= count; id++) {
      await Future<void>.delayed(const Duration(milliseconds: 10));
      yield (id: id, sensor: id.isEven ? 'temp' : 'humidity', value: 20.0 + id);
    }
  } finally {
    print('source closed'); // runs on normal end, error AND cancellation
  }
}

The try/finally is where you release resources: close a socket, stop a hardware reader, delete a temporary file. Dart guarantees the finally block runs when the subscriber cancels, which makes it the place for cleanup.

Stage 2: filter, enrich and batch

The standard Stream methods already cover filtering and one-at-a-time asynchronous mapping:

DART
Future<Event> calibrate(Event e) async {
  await Future<void>.delayed(const Duration(milliseconds: 5)); // pretend: a lookup
  return (id: e.id, sensor: e.sensor, value: e.value + 0.5);
}

final stream = sensorEvents()
    .where((e) => e.value < 28)   // drop high readings
    .asyncMap(calibrate);         // one event at a time, in order

asyncMap waits for each future before handling the next event, so order is preserved. If order does not matter and the lookups are independent, handle a whole batch concurrently instead (below).

Dart has no built-in "group into lists of N", but writing one as an extension method on Stream takes eight lines and shows the technique for any custom stage:

DART
extension Batching<T> on Stream<T> {
  Stream<List<T>> batch(int size) async* {
    var buffer = <T>[];
    await for (final item in this) {
      buffer.add(item);
      if (buffer.length == size) {
        yield buffer;
        buffer = <T>[];
      }
    }
    if (buffer.isNotEmpty) yield buffer; // flush the last, smaller batch
  }
}

Two points to notice. The last batch is flushed only when the upstream ends normally. And yield buffer; buffer = <T>[]; hands the finished list away and starts a fresh one, so the consumer never sees it change.

Here is the full pipeline in TypeScript. Source stages log when they clean up, so you can watch what take does to the stages behind it.

TypeScript Stages 1-2: a pipeline of generators (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or readFileSync(0)), it gets the next line from this box.

In scenario B only four events were produced even though the source could make ten, and [source] closed appeared exactly once, before the final batch. When take finished it closed the stage behind it, which closed the one behind that, down to the source. In Dart the equivalent is sensorEvents().asyncMap(calibrate).take(4).batch(3).

Stage 3: back-pressure

Suppose the source produces an event every 10 ms but the sink needs 40 ms per batch. If the source ignored the sink, events would pile up in memory, forever. Streams avoid that with a pull model, driven by the consumer:

DART
Future<void> main() async {
  final pipeline = sensorEvents()
      .where((e) => e.value < 28)
      .asyncMap(calibrate)
      .batch(3);

  await for (final batch in pipeline) {
    await Future<void>.delayed(const Duration(milliseconds: 40)); // slow sink
    print('wrote batch ${[for (final e in batch) e.id]}');
  }
}

await for asks for the next event only after the loop body has finished. While the body is busy, the listener is effectively paused, the pause travels back through batch, asyncMap and where, and the async* source stays suspended at its yield. The source does not run ahead, so memory stays flat no matter how slow the sink is.

sequenceDiagram
    participant K as Sink (await for)
    participant G as Source (async*)
    K->>G: give me the next event
    G-->>K: yield event 1
    Note over K: busy writing (40 ms)
    Note over G: suspended at yield
    K->>G: give me the next event
    G-->>K: yield event 2

The source only runs when the sink asks. Slow sink, slow source, and no growing buffer.

There is one way to lose this protection: a StreamController that you push into with add. The controller buffers everything you add while nobody is ready to receive it:

DART
final controller = StreamController<int>();
for (var i = 0; i < 1000000; i++) {
  controller.add(i); // all one million events wait in memory
}

When you must use a controller (to bridge a callback API, say), make it cooperative by reacting to demand with onPause and onResume, and by stopping production in onCancel:

DART
late StreamController<int> controller;
Timer? timer;
var next = 0;

controller = StreamController<int>(
  onListen: () => timer = Timer.periodic(
    const Duration(milliseconds: 10),
    (_) => controller.add(next++),
  ),
  onPause: () => timer?.cancel(),    // stop producing while paused
  onResume: () => timer = Timer.periodic(
    const Duration(milliseconds: 10),
    (_) => controller.add(next++),
  ),
  onCancel: () => timer?.cancel(),   // clean up when the listener leaves
);

Prefer an async* generator whenever you can. It gives you back-pressure for free, while a controller makes you earn it.

The next stand-in contrasts the two models. The generator only produces when the loop asks for another event. The eager version produces everything first.

TypeScript Stage 3: pull-based versus eager production (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or readFileSync(0)), it gets the next line from this box.

Stage 4: cancellation

Pipelines start and they must also stop: the user presses a button, a timeout fires, the program shuts down. Dart gives you three levels.

Stop listening. A StreamSubscription has a cancel() method. Leaving an await for early with break, return or an exception cancels the subscription for you.

DART
final subscription = pipeline.listen((batch) => print('batch ${batch.length}'));
await Future<void>.delayed(const Duration(milliseconds: 100));
await subscription.cancel(); // the source's finally block runs: "source closed"

Two details bite beginners. After cancel() the onDone callback does not run, so if other code waits for "the pipeline is done", complete a Completer yourself in your stop method. And after cancellation, a stage cannot yield anymore, so a half-filled batch is simply dropped. Only cleanup belongs in finally.

Limit the stream. take(n), takeWhile(test) and timeout(duration) end a stream by count, condition or silence:

DART
final firstHundred = sensorEvents(count: 1000000).take(100);
final guarded = sensorEvents().timeout(
  const Duration(seconds: 2),
  onTimeout: (sink) => sink.close(), // end the stream if nothing arrives for 2 s
);

Ask the source to finish (graceful stop). A hard cancel throws away work in flight. Often you prefer to tell the source "no more events" and let everything already inside the pipeline drain, so the last partial batch is flushed. A Completer works as a stop signal, and Future.any wakes a sleeping producer immediately instead of after its next delay:

DART
class StopToken {
  final _completer = Completer<void>();
  bool get isStopped => _completer.isCompleted;
  Future<void> get whenStopped => _completer.future;

  void stop() {
    if (!_completer.isCompleted) _completer.complete();
  }
}

Stream<int> ticker(StopToken token, Duration period) async* {
  var id = 0;
  try {
    while (!token.isStopped) {
      await Future.any([Future<void>.delayed(period), token.whenStopped]);
      if (token.isStopped) break;
      yield ++id;
    }
  } finally {
    print('source cleaned up');
  }
}

The last stand-in shows both ways to stop the same pipeline, plus the "wake a sleeping source" trick, using JavaScript's AbortController as the stop token.

TypeScript Stage 4: graceful stop, hard stop and waking a sleeper (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or readFileSync(0)), it gets the next line from this box.

Putting it together: a Pipeline class

The pieces fit into one small class that hides the stream plumbing and offers start, stop and done. It is the pattern you will use for services, workers and long-running tasks:

DART
class Pipeline {
  Pipeline(this._source);

  final Stream<Event> _source;
  StreamSubscription<List<Event>>? _subscription;
  final _finished = Completer<void>();

  Future<void> get done => _finished.future;

  void start(void Function(List<Event>) onBatch) {
    _subscription = _source
        .where((e) => e.value < 28)
        .asyncMap(calibrate)
        .batch(3)
        .listen(
          onBatch,
          onError: (Object error) => print('stage failed: $error'),
          onDone: _complete,
        );
  }

  Future<void> stop() async {
    await _subscription?.cancel();
    _complete(); // onDone does not run after cancel
  }

  void _complete() {
    if (!_finished.isCompleted) _finished.complete();
  }
}

Errors travel through the stream too. An exception thrown inside calibrate becomes an error event: with listen(..., onError:) the pipeline keeps going (unless you pass cancelOnError: true), while await for ends at the first error. Decide on purpose which behaviour you want, and handle an expected failure (one bad sensor reading) inside the stage so it does not stop the others.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does an async* source not flood memory when the sink is slow?

    Streams are pull-based. await for requests the next event only when the loop body is done, so the generator stays suspended at its yield.
  2. 2. When does a StreamController buffer unboundedly?

    A controller stores whatever you add while no one is ready. Use onPause and onResume to stop producing, or prefer an async* generator.
  3. 3. What happens to onDone after you call subscription.cancel()?

    Cancelling detaches your callbacks, including onDone. A stop method should therefore complete whatever the rest of the program waits on.
  4. 4. What is the difference between a hard cancel and a graceful stop?

    Cancelling the subscription stops delivery at once, so a partly filled batch is lost. Signalling the source to finish lets downstream stages complete and flush normally.
  5. 5. Why use Future.any([delay, token.whenStopped]) in a polling source?

    Future.any completes with whichever future finishes first, so a stop signal wakes the producer immediately.

Summary

  • A pipeline is a chain of functions from stream to stream: where, asyncMap, take, and your own stages written as async* generators or extension methods.
  • Streams are pull-based: await for and asyncMap consume one event at a time, and pauses travel back to the source, which is back-pressure.
  • A StreamController buffers whatever you push; make it cooperative with onPause, onResume and onCancel, or use an async* generator.
  • Put cleanup in finally: it runs on completion, errors and cancellation. A cancelled stage cannot yield, so partial batches are dropped.
  • Stop a pipeline by cancelling the subscription (hard), limiting it with take or timeout, or signalling the source to finish (graceful); after cancel() complete any "done" Completer yourself.

Next: Project: Typed REST Client.