Skip to content
Tutorials Center
My learning

Topics

C# Reference ยท Async and Concurrency

TPL Dataflow

Reference for TPL Dataflow in C#: ActionBlock, TransformBlock, BufferBlock, BatchBlock, LinkTo, bounded capacity, parallelism and completion propagation.

TPL Dataflow (System.Threading.Tasks.Dataflow) is a library for in-process pipelines. You create blocks, each of which receives messages, processes them and optionally passes results on. You connect blocks with LinkTo, post messages into the first block and the framework moves data between them, running each block's work on the thread pool.

Compared with hand-written queues and threads, blocks give you buffering, ordering, bounded capacity (backpressure), configurable parallelism and clean completion handling. A typical pipeline is: parse, then transform, then batch, then save. On our runner the library is available without extra packages; in your own project you may need the System.Threading.Tasks.Dataflow NuGet package.

C#
using System.Threading.Tasks.Dataflow;

var square = new TransformBlock<int, int>(n => n * n);
var print = new ActionBlock<int>(n => Console.WriteLine(n));
square.LinkTo(print, new DataflowLinkOptions { PropagateCompletion = true });

Try it

C# A two-stage pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

C# Batching, parallelism and backpressure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Syntax

The most used block types:

Block Description
ActionBlock<TInput> Runs a delegate (sync or async) for each message. A target only: it has no output.
TransformBlock<TInput, TOutput> Runs a function per message and offers the results to linked blocks. Output order matches input order.
TransformManyBlock<TInput, TOutput> Like TransformBlock but returns zero or more outputs per input.
BufferBlock<T> A queue; messages go out first in, first out.
BroadcastBlock<T> Keeps the latest value and offers a copy to every linked target.
BatchBlock<T> Collects items and emits T[] of a fixed size.
JoinBlock, WriteOnceBlock Combine inputs, or accept one value only.

Key members and options:

C#
IDisposable LinkTo(ITargetBlock<T> target, DataflowLinkOptions options);
bool Post(T item);                              // false if the block is full or completed
Task<bool> SendAsync(T item);                   // waits for room
void Complete();                                // no more input
Task Completion { get; }                        // finishes when everything is processed

new ExecutionDataflowBlockOptions
{
    MaxDegreeOfParallelism = 4,   // default 1
    BoundedCapacity = 100,        // default unbounded (-1)
    CancellationToken = token,
}

Errors: an exception thrown by a block's delegate faults that block. Its Completion task becomes faulted, and the block stops accepting messages. With PropagateCompletion = true the fault travels downstream, wrapped in an AggregateException (nested once per hop).

Usage notes

  • Always set PropagateCompletion = true on links, otherwise calling Complete() on the head does not finish the later blocks and await tail.Completion waits forever.
  • Default parallelism is 1, so a block processes messages one at a time and in order. Raise MaxDegreeOfParallelism only if the work is independent; TransformBlock still emits in input order.
  • Set BoundedCapacity for backpressure. Without it an unbounded buffer can eat memory when a producer outruns a consumer. With a bound, prefer await SendAsync(item) over Post, which just returns false.
  • Bounded blocks and BoundedCapacity interact with linking: a fast upstream block with a small downstream capacity naturally slows to match.
  • Use DataflowLinkOptions.Append, filters (LinkTo(target, predicate)) and a catch-all link for routing. A message that no link accepts stays in the block and can clog it, so add a final link to a null sink (DataflowBlock.NullTarget<T>()).
  • Dataflow versus Channel<T>: channels are a simpler primitive for producer-consumer. Dataflow adds ready-made stages, batching and propagation, but is heavier. Choose channels for one queue, dataflow for multi-stage graphs.
  • Cancellation goes through CancellationToken in the block options. Canceled blocks drop their queued messages.