Skip to content
Tutorials Center
My learning

Topics

Lesson 100 of 182 ยท Async and Concurrency

Parallel and PLINQ

Learn data parallelism in C# - Parallel.For, Parallel.ForEach, Parallel.Invoke, ForEachAsync, PLINQ with AsParallel and AsOrdered, partitioning, and when parallel code is slower.

Tasks let you run different jobs at the same time. Data parallelism is a different idea: you have one job that repeats over many items (resize 10,000 images, score a million rows, check a big range of numbers), and you want to spread those items across all the CPU cores. .NET has two tools for this: the Parallel class and PLINQ (Parallel LINQ). Both take a loop or a query you already understand and run its iterations on several threads from the thread pool.

Parallel code is easy to start and easy to get wrong. A parallel loop is only correct when its iterations do not disturb each other, and only fast when the work per item is large enough to pay for the splitting. This lesson shows how to use both tools, how to collect results safely and deterministically, and when to leave a loop alone.

Our editor runs your code on a server with one CPU, so a parallel loop here cannot run faster than a plain loop; you will not see a speed-up. That is fine for learning the API: every example is written so its printed result is the same on one core or sixty-four. Never quote timings from the sandbox, and measure on your own machine before you decide that parallel code helps.

CPU-bound versus I/O-bound

Parallel loops are for CPU-bound work: the thread is busy calculating, so more cores means more calculations per second. If the work is I/O-bound (waiting for a web server, a disk or a database), extra threads just sit and wait. For that kind of work you use async/await and Task.WhenAll, or Parallel.ForEachAsync, shown later on this page.

Parallel.For

Parallel.For(from, to, body) is the parallel version of a for loop. The range includes from and excludes to, exactly like for (int i = from; i < to; i++). The body runs once per index, on whichever thread the scheduler picks, in no guaranteed order. The call itself blocks until every iteration has finished.

C# Parallel.For with one slot per iteration

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

This is the safest shape for a parallel loop: iteration i reads and writes only element i. Two iterations never touch the same memory, so no locks are needed and the result is always 0, 1, 4, 9, ... no matter in which order the iterations ran.

Shared variables are the classic bug

The tempting mistake is to accumulate into one shared variable:

C#
long total = 0;
Parallel.For(1, 1_000_001, i =>
{
    total += i;     // BUG: many threads read-modify-write the same variable
});

total += i is really three steps (read, add, write). When two threads interleave them, one update overwrites the other and the final sum is too small. It may even look right on a quiet machine, which is what makes it dangerous. Wrap it in a lock and every iteration queues up for the lock, which throws away the parallel speed-up. The better answer is to give each thread its own running total and combine them once at the end.

Thread-local state: the localInit overload

One Parallel.For overload does exactly that. It takes three delegates: localInit creates a private value for each worker, the body updates it without any synchronization, and localFinally runs once per worker to merge that private value into the shared one. Only the merge needs a lock (or an Interlocked call, see lock, Lock and Interlocked).

C# Thread-local totals merged at the end

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The body returns the new subtotal, so the loop never writes to shared memory until the very end. Addition is associative (the grouping does not change the answer), which is why combining the per-worker subtotals in any order gives the same total. Floating-point addition is not perfectly associative, so a parallel sum of double values can differ in the last digits from run to run.

Parallel.ForEach

Parallel.ForEach does the same for any IEnumerable<T>: it pulls items from the sequence and hands them to workers. The sequence is read by several threads at once, so .NET protects the enumerator for you, but your body must still be safe.

When the order of results matters, do not append to a shared List<T> from the body (List<T> is not thread-safe). Preallocate an array and write by index, or use a concurrent collection (see Concurrent Collections) and sort afterwards.

C# ForEach with results written by index

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

ParallelOptions controls the loop. MaxDegreeOfParallelism caps how many iterations run at once (the default lets the scheduler decide, usually the number of cores). It also accepts a CancellationToken, so a loop can be stopped from outside; see Cancellation and Timeouts. Lowering the cap is useful when each iteration is heavy on memory or on a shared resource such as a database connection pool.

Stopping early: Break and Stop

The body can take a ParallelLoopState parameter. state.Stop() asks the loop to start no new iterations as soon as possible. state.Break() is gentler: iterations before the current index still finish, which matters when you want the lowest matching index. Both only prevent iterations from starting; iterations already running will complete, so you may still see some work done after the call.

Parallel.Invoke

When you have a handful of different jobs rather than a repeated one, Parallel.Invoke runs several actions concurrently and returns when all are done.

C# Parallel.Invoke runs different jobs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Each action writes to its own variable, so there is no sharing. After Invoke returns, all three variables are safely visible to the main thread.

What happens when an iteration throws

If an iteration throws, the loop stops starting new iterations, waits for those already running, and then throws an AggregateException that holds every exception thrown by any iteration. Your catch has to look inside.

C# Exceptions are bundled in AggregateException

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

How many exceptions are collected depends on timing: once the first failure is seen, the loop stops scheduling new iterations, so you may get one failure or all three. So the example prints only facts that are always true; your own code should never depend on the count. Because exceptions can come from several threads, sort or group them before you display them if you want stable output.

Parallel.ForEachAsync

Since .NET 6 there is Parallel.ForEachAsync for asynchronous bodies. It runs up to MaxDegreeOfParallelism async iterations at the same time and, unlike Parallel.ForEach, never blocks threads while they wait. This is the right tool for "call this API for each of 500 ids, but at most 8 at a time".

C# ForEachAsync with a concurrency limit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The body receives a CancellationToken as its second parameter; pass it on to the calls you make. Completion order is arbitrary, so the example sorts by key before printing.

PLINQ: parallel queries

LINQ becomes parallel with one call: AsParallel(). After it, Where, Select, Count, Sum and friends run on several threads, and the query still reads the same.

C# A PLINQ query

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Aggregations such as Count and Sum give exact answers for integers, because the partial results are combined correctly.

Ordering in PLINQ

A normal LINQ query returns items in the source order. A parallel query does not: threads finish in any order and the results come out in whatever order they are ready. If you need the original order, add AsOrdered(). That makes PLINQ buffer and reorder the results, which costs time and memory, so use it only when order matters. If you only need a sorted result, an ordinary OrderBy at the end is clearer.

C# AsOrdered keeps the source order

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Without either AsOrdered or OrderBy, ToArray() could return 30, 60, 90, ... in a different order from run to run. Other useful PLINQ methods are WithDegreeOfParallelism(n), WithCancellation(token), and ForAll(action), which runs an action on each result on the worker threads without merging them back first. Like any parallel loop body, an action given to ForAll must be thread-safe.

Partitioning: how the work is split

Both tools cut the input into partitions and give each worker one at a time. For arrays and lists, whose length and indexing are known, .NET can split cheaply into ranges. For a plain IEnumerable<T> it has to hand out items in small chunks. Two rules follow from this:

  • Uneven work needs small chunks. If item 1 takes a second and the rest take a millisecond, a big fixed range leaves one worker busy while the others idle. The default partitioners adapt, handing out larger chunks over time.
  • Tiny work needs big chunks. If the body is x[i] = x[i] * 2, the cost of calling a delegate per item dwarfs the work itself. Process ranges instead (use Partitioner.Create(0, n) and loop inside each range) or simply do not parallelize.

When parallelism makes things slower

A parallel loop adds costs: creating workers, partitioning, calling a delegate per item, merging, and sometimes cache contention. It wins only when the total work is large and the iterations are independent. Skip it when:

  • the body is a few nanoseconds of work (adding two numbers, reading a field);
  • the whole loop takes under a millisecond or so;
  • the work is I/O-bound (use async instead);
  • the iterations must share state or run in order;
  • you are inside a web request handler: parallel loops take pool threads from other requests, and under load that makes the server slower overall.

Measure first. Write the sequential version, make sure it is correct, then try the parallel one on realistic data and compare. A common finding is a good speed-up for heavy per-item work on large inputs and no benefit at all for everything else.

If the loop body is non-trivial, think of the "one slot per iteration" shape as your default: compute independent results into an array indexed by iteration number. It is correct, lock-free and keeps the output order stable.

Determinism checklist

Parallel results are reproducible only if you deliberately make them so:

  1. Do not rely on iteration order, thread ids or finishing order.
  2. Write to separate memory per iteration, or merge thread-local results at the end.
  3. Use AsOrdered or an explicit sort when order matters.
  4. Use integer or decimal arithmetic for sums you compare exactly; floating-point sums may differ slightly.
  5. Never share a Random instance between threads (Random.Shared is thread-safe), and remember that a seeded Random per worker gives different numbers per split.
  6. Catch AggregateException and treat the number of failures as unpredictable.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which kind of work benefits from Parallel.For?

    Parallel loops spread independent CPU work over cores. I/O waits and dependent iterations do not benefit.
  2. 2. Why is total += i inside a Parallel.For body wrong?

    The update is read, add, write. Threads interleave those steps and overwrite each other, giving a total that is too small.
  3. 3. What does a parallel loop throw when one or more iterations fail?

    Exceptions from worker threads are collected and rethrown together as an AggregateException.
  4. 4. A PLINQ query must return items in the original order. What do you add?

    AsOrdered makes PLINQ preserve source order at some cost. Without it the result order is not guaranteed.
  5. 5. Which tool is best for calling an API for each of 500 ids, at most 8 at a time?

    ForEachAsync runs async bodies with a concurrency limit and does not block threads while waiting.

Summary

  • Parallel.For, Parallel.ForEach, Parallel.Invoke and PLINQ's AsParallel() spread CPU-bound work across cores; async I/O belongs to await and Parallel.ForEachAsync.
  • Iterations must be independent. Write results to separate slots, or use thread-local state (localInit/localFinally) and merge once; never accumulate into a shared variable.
  • Iteration order and finishing order are not guaranteed; sort, index, or use AsOrdered() to get stable output.
  • Exceptions arrive wrapped in an AggregateException; ParallelOptions sets the degree of parallelism and a cancellation token.
  • Parallelism has overhead and can be slower for small or I/O-bound work. Write the sequential version first and measure.

Next: lock, Lock and Interlocked