C# Reference ยท Async and Concurrency
TPL Dataflow
Reference for TPL Dataflow in C#: ActionBlock, TransformBlock, BufferBlock, BatchBlock, LinkTo, bounded capacity, parallelism and completion propagation.
TPL Dataflow (System.Threading.Tasks.Dataflow) is a library for in-process pipelines. You create blocks, each of which receives messages, processes them and optionally passes results on. You connect blocks with LinkTo, post messages into the first block and the framework moves data between them, running each block's work on the thread pool.
Compared with hand-written queues and threads, blocks give you buffering, ordering, bounded capacity (backpressure), configurable parallelism and clean completion handling. A typical pipeline is: parse, then transform, then batch, then save. On our runner the library is available without extra packages; in your own project you may need the System.Threading.Tasks.Dataflow NuGet package.
using System.Threading.Tasks.Dataflow; var square = new TransformBlock<int, int>(n => n * n); var print = new ActionBlock<int>(n => Console.WriteLine(n)); square.LinkTo(print, new DataflowLinkOptions { PropagateCompletion = true });
Try it
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.
Syntax
The most used block types:
| Block | Description |
|---|---|
ActionBlock<TInput> |
Runs a delegate (sync or async) for each message. A target only: it has no output. |
TransformBlock<TInput, TOutput> |
Runs a function per message and offers the results to linked blocks. Output order matches input order. |
TransformManyBlock<TInput, TOutput> |
Like TransformBlock but returns zero or more outputs per input. |
BufferBlock<T> |
A queue; messages go out first in, first out. |
BroadcastBlock<T> |
Keeps the latest value and offers a copy to every linked target. |
BatchBlock<T> |
Collects items and emits T[] of a fixed size. |
JoinBlock, WriteOnceBlock |
Combine inputs, or accept one value only. |
Key members and options:
IDisposable LinkTo(ITargetBlock<T> target, DataflowLinkOptions options); bool Post(T item); // false if the block is full or completed Task<bool> SendAsync(T item); // waits for room void Complete(); // no more input Task Completion { get; } // finishes when everything is processed new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 4, // default 1 BoundedCapacity = 100, // default unbounded (-1) CancellationToken = token, }
Errors: an exception thrown by a block's delegate faults that block. Its Completion task becomes faulted, and the block stops accepting messages. With PropagateCompletion = true the fault travels downstream, wrapped in an AggregateException (nested once per hop).
Usage notes
- Always set
PropagateCompletion = trueon links, otherwise callingComplete()on the head does not finish the later blocks andawait tail.Completionwaits forever. - Default parallelism is 1, so a block processes messages one at a time and in order. Raise
MaxDegreeOfParallelismonly if the work is independent;TransformBlockstill emits in input order. - Set
BoundedCapacityfor backpressure. Without it an unbounded buffer can eat memory when a producer outruns a consumer. With a bound, preferawait SendAsync(item)overPost, which just returnsfalse. - Bounded blocks and
BoundedCapacityinteract with linking: a fast upstream block with a small downstream capacity naturally slows to match. - Use
DataflowLinkOptions.Append, filters (LinkTo(target, predicate)) and a catch-all link for routing. A message that no link accepts stays in the block and can clog it, so add a final link to a null sink (DataflowBlock.NullTarget<T>()). - Dataflow versus
Channel<T>: channels are a simpler primitive for producer-consumer. Dataflow adds ready-made stages, batching and propagation, but is heavier. Choose channels for one queue, dataflow for multi-stage graphs. - Cancellation goes through
CancellationTokenin the block options. Canceled blocks drop their queued messages.