C# Reference ยท Files and I/O
System.IO.Pipelines
Reference for System.IO.Pipelines in C#: Pipe, PipeReader, PipeWriter, ReadOnlySequence parsing, backpressure and when to prefer pipelines over streams.
System.IO.Pipelines is a library for fast, low-allocation parsing of streaming data, such as network protocols. A Pipe connects a producer and a consumer: the producer asks a PipeWriter for memory, fills it and calls FlushAsync; the consumer awaits PipeReader.ReadAsync, looks at the bytes that have arrived as a ReadOnlySequence<byte>, and tells the reader how much it consumed and how much it examined.
Compared with a plain Stream, a pipe manages the buffers for you (pooled memory, no copying into your own arrays), lets you leave a half-received message in the buffer until the rest arrives, and applies backpressure so a fast producer waits when the consumer lags. Kestrel, the ASP.NET Core web server, is built on this library.
var pipe = new Pipe(); await pipe.Writer.WriteAsync("hello\n"u8.ToArray()); pipe.Writer.Complete(); ReadResult result = await pipe.Reader.ReadAsync();
Try it
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.
Syntax
public class Pipe { public Pipe(); public Pipe(PipeOptions options); public PipeReader Reader { get; } public PipeWriter Writer { get; } public void Reset(); } public abstract class PipeReader { public abstract ValueTask<ReadResult> ReadAsync(CancellationToken cancellationToken = default); public abstract bool TryRead(out ReadResult result); public abstract void AdvanceTo(SequencePosition consumed, SequencePosition examined); public abstract void Complete(Exception? exception = null); public static PipeReader Create(Stream stream, StreamPipeReaderOptions? options = null); } public abstract class PipeWriter { public abstract Memory<byte> GetMemory(int sizeHint = 0); public abstract Span<byte> GetSpan(int sizeHint = 0); public abstract void Advance(int bytes); public abstract ValueTask<FlushResult> FlushAsync(CancellationToken cancellationToken = default); public abstract void Complete(Exception? exception = null); public static PipeWriter Create(Stream stream, StreamPipeWriterOptions? options = null); }
| Member | Description |
|---|---|
GetMemory / GetSpan + Advance |
The writer hands out a buffer, you fill part of it, then you report how many bytes you wrote. WriteAsync(ReadOnlyMemory<byte>) is a one-call shortcut that also flushes. |
ReadAsync |
Completes when new data arrives or the writer completes. Returns a ReadResult with Buffer, IsCompleted and IsCanceled. |
AdvanceTo(consumed, examined) |
consumed data is released. Data between consumed and examined is kept, and the next ReadAsync waits for more data than that. |
PipeOptions |
pauseWriterThreshold, resumeWriterThreshold, minimumSegmentSize, MemoryPool and schedulers. |
Complete / CompleteAsync |
Signals that no more data will be written (writer) or read (reader). |
Errors: InvalidOperationException when you call ReadAsync twice without AdvanceTo, or AdvanceTo with positions outside the buffer; an exception passed to Complete is rethrown on the other side.
Usage notes
- Always call
AdvanceToafter everyReadAsync. Even if you consumed nothing, passbuffer.Startasconsumedandbuffer.Endasexamined; otherwise the pipe thinks you are still reading and the next read throws. ReadOnlySequence<byte>can span several segments. Usebuffer.IsSingleSegmentandbuffer.FirstSpanas a fast path, andSequenceReader<byte>for parsing across segment boundaries.- Never keep the buffer after
AdvanceTo. Its memory goes back to the pool. Copy what you need (ToArray(),Encoding.GetString) first. - Backpressure is built in. When unread bytes exceed
pauseWriterThreshold(default 64 KB),FlushAsyncdoes not complete until the reader drops belowresumeWriterThreshold. This keeps memory bounded. - Complete both ends. If the reader never completes, the writer can stay suspended; if the writer never completes, the reader never sees
IsCompleted. - Not always the right tool. For simple line-by-line file processing
StreamReader.ReadLineAsyncis easier and fast enough. Pipelines pay off for protocol parsers, servers and anything handling very large numbers of connections. - Testing. Since a
Piperuns entirely in memory, parsers can be tested without sockets by writing sample bytes into the writer. - Packaging. The library is part of the .NET shared framework in current versions; older projects referenced the
System.IO.PipelinesNuGet package.