Skip to content
Tutorials Center
My learning

Topics

C# Reference ยท Files and I/O

System.IO.Pipelines

Reference for System.IO.Pipelines in C#: Pipe, PipeReader, PipeWriter, ReadOnlySequence parsing, backpressure and when to prefer pipelines over streams.

System.IO.Pipelines is a library for fast, low-allocation parsing of streaming data, such as network protocols. A Pipe connects a producer and a consumer: the producer asks a PipeWriter for memory, fills it and calls FlushAsync; the consumer awaits PipeReader.ReadAsync, looks at the bytes that have arrived as a ReadOnlySequence<byte>, and tells the reader how much it consumed and how much it examined.

Compared with a plain Stream, a pipe manages the buffers for you (pooled memory, no copying into your own arrays), lets you leave a half-received message in the buffer until the rest arrives, and applies backpressure so a fast producer waits when the consumer lags. Kestrel, the ASP.NET Core web server, is built on this library.

C#
var pipe = new Pipe();
await pipe.Writer.WriteAsync("hello\n"u8.ToArray());
pipe.Writer.Complete();
ReadResult result = await pipe.Reader.ReadAsync();

Try it

C# Parsing lines from a pipe

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

C# Backpressure and wrapping a stream

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Syntax

C#
public class Pipe
{
    public Pipe();
    public Pipe(PipeOptions options);
    public PipeReader Reader { get; }
    public PipeWriter Writer { get; }
    public void Reset();
}

public abstract class PipeReader
{
    public abstract ValueTask<ReadResult> ReadAsync(CancellationToken cancellationToken = default);
    public abstract bool TryRead(out ReadResult result);
    public abstract void AdvanceTo(SequencePosition consumed, SequencePosition examined);
    public abstract void Complete(Exception? exception = null);
    public static PipeReader Create(Stream stream, StreamPipeReaderOptions? options = null);
}

public abstract class PipeWriter
{
    public abstract Memory<byte> GetMemory(int sizeHint = 0);
    public abstract Span<byte> GetSpan(int sizeHint = 0);
    public abstract void Advance(int bytes);
    public abstract ValueTask<FlushResult> FlushAsync(CancellationToken cancellationToken = default);
    public abstract void Complete(Exception? exception = null);
    public static PipeWriter Create(Stream stream, StreamPipeWriterOptions? options = null);
}
Member Description
GetMemory / GetSpan + Advance The writer hands out a buffer, you fill part of it, then you report how many bytes you wrote. WriteAsync(ReadOnlyMemory<byte>) is a one-call shortcut that also flushes.
ReadAsync Completes when new data arrives or the writer completes. Returns a ReadResult with Buffer, IsCompleted and IsCanceled.
AdvanceTo(consumed, examined) consumed data is released. Data between consumed and examined is kept, and the next ReadAsync waits for more data than that.
PipeOptions pauseWriterThreshold, resumeWriterThreshold, minimumSegmentSize, MemoryPool and schedulers.
Complete / CompleteAsync Signals that no more data will be written (writer) or read (reader).

Errors: InvalidOperationException when you call ReadAsync twice without AdvanceTo, or AdvanceTo with positions outside the buffer; an exception passed to Complete is rethrown on the other side.

Usage notes

  • Always call AdvanceTo after every ReadAsync. Even if you consumed nothing, pass buffer.Start as consumed and buffer.End as examined; otherwise the pipe thinks you are still reading and the next read throws.
  • ReadOnlySequence<byte> can span several segments. Use buffer.IsSingleSegment and buffer.FirstSpan as a fast path, and SequenceReader<byte> for parsing across segment boundaries.
  • Never keep the buffer after AdvanceTo. Its memory goes back to the pool. Copy what you need (ToArray(), Encoding.GetString) first.
  • Backpressure is built in. When unread bytes exceed pauseWriterThreshold (default 64 KB), FlushAsync does not complete until the reader drops below resumeWriterThreshold. This keeps memory bounded.
  • Complete both ends. If the reader never completes, the writer can stay suspended; if the writer never completes, the reader never sees IsCompleted.
  • Not always the right tool. For simple line-by-line file processing StreamReader.ReadLineAsync is easier and fast enough. Pipelines pay off for protocol parsers, servers and anything handling very large numbers of connections.
  • Testing. Since a Pipe runs entirely in memory, parsers can be tested without sockets by writing sample bytes into the writer.
  • Packaging. The library is part of the .NET shared framework in current versions; older projects referenced the System.IO.Pipelines NuGet package.