Skip to content
Tutorials Center
My learning

Topics

Lesson 175 of 182 ยท Projects

Project: Background Job Queue

Build a background job queue in C# in four stages: a bounded Channel with backpressure, retries with exponential backoff, a BackgroundService worker pool, and timeouts, shutdown and tests.

Some work should not happen while a user waits: sending an email, resizing an image, generating a report, calling a flaky partner API. A web request puts a job on a queue and returns immediately; workers in the background take jobs from the queue and do them. Almost every real system has this shape, and building one teaches the most important habits of server-side C#: bounded buffers and backpressure, retries with backoff, cancellation and timeouts, a clean shutdown, and tests for code that normally depends on the clock.

You build the queue in four stages, and every stage is a complete program. In stage 1 you build the core, a bounded Channel<T>, and see what backpressure means. In stage 2 you write a job runner that retries transient failures with exponential backoff and jitter, gives up on permanent failures, and records everything, without a single real sleep in the demo. In stage 3 you host a pool of workers in a BackgroundService with dependency injection and options. In stage 4 you add per-job timeouts and graceful shutdown, and write a test suite for the whole system.

Production systems often use a ready-made job library or a message broker (Hangfire, Quartz.NET, Azure Service Bus, RabbitMQ) when jobs must survive a crash. An in-memory queue loses its jobs when the process dies. We build the in-memory kind because it is the best way to learn the mechanics, and because it is genuinely the right tool for work that can be retried from its source.

The plan

Stage New idea Tools
1 A bounded buffer between producers and a consumer Channel<T>, async streams
2 Failure is normal: retry, back off, give up exceptions, exception filters, Task.Delay
3 Run a worker pool for the life of the application BackgroundService, dependency injection, options
4 Time limits, shutdown and tests cancellation, linked tokens, disposal

Stage 1: a bounded channel

A Channel<T> is a thread-safe pipe: any number of producers write, any number of consumers read, and items come out in the order they went in. It replaces the old BlockingCollection<T> for asynchronous code, because waiting for an item or for free space uses await instead of blocking a thread (channels).

The most important decision is the capacity:

  • An unbounded channel never makes a producer wait. If producers are faster than consumers, the queue grows without limit, until the process runs out of memory. This is the most common way a queue takes a service down.
  • A bounded channel has a fixed capacity. When it is full, the producer must wait (WriteAsync does not complete until there is room), or be refused (TryWrite returns false), or the oldest/newest item is dropped (other FullMode choices). Slowing the producers down to the speed of the consumers is called backpressure, and it is what keeps a system stable under load. A web endpoint can turn a refusal into 503 Service Unavailable with a Retry-After header, which is far better than falling over.

The demo has two parts. First, nobody is reading, so a channel with capacity 3 accepts three jobs and refuses the rest. Second, a single consumer processes jobs while a fast producer writes eight of them: the producer is forced to wait whenever the queue is full, and we record the queue depth after each write to prove it never exceeded the capacity.

Finishing is part of the design. The producer calls writer.Complete() to say "no more items"; the consumer's await foreach loop drains what is left and then ends by itself. No poison-pill items, no flags.

C# Stage 1: a bounded Channel, backpressure and completion

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

What to take from it:

  • TryWrite never waits; WriteAsync waits for space. Choose by what the caller can do. A web request cannot wait for minutes, so it uses TryWrite and reports failure; a batch importer can wait, so it uses WriteAsync.
  • SingleReader and SingleWriter are promises that let the channel use a cheaper internal implementation. Only set them when they are true; if two tasks read a channel declared with SingleReader = true, the behaviour is undefined.
  • ReadAllAsync() is an IAsyncEnumerable<T>. The await foreach loop waits for items without blocking a thread, and ends when the channel is completed and empty (async streams).
  • Completion is the shutdown signal. Complete() lets all items already in the channel be delivered. After it, writes fail (TryWrite returns false; WriteAsync throws ChannelClosedException).
  • Items come out in FIFO order, which is why the processed list is deterministic with a single consumer. With several consumers, start order is still FIFO, but completion order depends on how long each job takes.

Do not use Task.Run to fire and forget a job from a request handler (_ = Task.Run(() => SendEmail())). The work is invisible, unbounded and lost on shutdown, and exceptions disappear. A queue with workers makes the same thing visible, limited and stoppable.

Stage 2: failures, retries and backoff

Real jobs fail. The important question is how, because the right reaction differs:

  • A transient failure goes away by itself: a network timeout, a database deadlock, a service that is restarting. Retrying helps.
  • A permanent failure never goes away: bad input, a missing file, a bug (NullReferenceException). Retrying wastes time and makes noise. The job should fail at once and be recorded.

We model the difference with an exception type, TransientException, which jobs throw when they know an error is worth retrying. (A real system often classifies by type, for example HttpRequestException with status 503 is transient and 404 is permanent.) Everything else is permanent.

Exponential backoff with jitter

When a dependency is struggling, hammering it with immediate retries makes things worse. Exponential backoff waits longer after each failure: 100 ms, 200 ms, 400 ms, 800 ms, up to a maximum. Jitter adds randomness, so a thousand jobs that failed at the same moment don't all retry at the same moment either. We use "equal jitter": the delay is between half and the full computed value.

Testing time

Code that sleeps is slow to test and hard to assert on. The cure is to treat "wait for this long" as a dependency: JobRunner takes a delay function. In production it is Task.Delay. In the demo and tests it is a fake that records the requested delay and returns immediately, so the program finishes at once and we can print exactly what the runner asked for. (.NET also has TimeProvider for the same purpose with the clock itself; see the to-do project.)

The runner takes a CancellationToken as well. When the token fires, either during the job or during a backoff wait, the result is Cancelled and no retry is attempted: cancellation always wins over retrying.

C# Stage 2: a retrying job runner with a fake clock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Study the runner, because this loop is the heart of the project:

  • The for (;;) loop with returns is easier to reason about than recursion or a while (retries > 0) counter. Each iteration is one attempt; every way out of the method is an explicit return with a result.
  • Catch order is decision order. Cancellation by the caller is checked first (an exception filter when (stopping.IsCancellationRequested)), then cancellation by our own per-attempt timeout, then TransientException, then everything else. An exception filter decides whether to catch without unwinding, so a filter that says no leaves the exception for the next handler or the caller (exception filters).
  • A linked token source per attempt. CreateLinkedTokenSource(stopping) makes a token that fires when the outer token fires or when we call CancelAfter. That is how one job gets a private timeout (stage 4) without the outer token knowing. It must be disposed (the using), or timers leak.
  • The tuple deconstruction (failure, error, retryable) = (...) assigns three variables at once, so the single decision after the try can read all three. Definite assignment still holds: every catch path assigns all of them or returns.
  • Attempts are counted from 1, delays from the second attempt. DelayBefore(2) is the wait before the first retry. Check the no-jitter table: 100, 200, 400, 800, 1600, then capped at 2000 ms. Without a cap the twelfth retry would wait over three minutes.
  • MaxAttempts: 1 means "never retry". The job one-shot throws a transient error but is not retried because the budget is spent.
  • The fake delay proves the behaviour. For flaky the runner asked for waits of 100 ms and 200 ms (before attempts 2 and 3) and then succeeded. For buggy it asked for no waits at all: permanent errors are not retried.
  • Random is not thread safe. The jittered policy here uses a seeded Random so the table is reproducible, which is fine because this demo is single-threaded. In stage 3, where many workers run at once, the policy is given Random.Shared, which is safe for concurrent use.

Retrying has a classic trap: a retried job runs twice if the first attempt actually succeeded but the reply was lost. Jobs that are retried should be idempotent, which means running them twice has the same effect as once (use a unique key to detect "already done", upsert instead of insert, send emails with a message id). Design for it from the start; it is very hard to add later.

Stage 3: a worker pool in a BackgroundService

Now we run the queue as part of an application. ASP.NET Core and worker-service applications are built on the generic host (Host.CreateApplicationBuilder). The host starts everything registered with AddHostedService<T>(), waits, and on shutdown asks each service to stop. BackgroundService is the standard base class for a long-running service: you override ExecuteAsync(CancellationToken stoppingToken), and the token is cancelled when the host begins to shut down (generic host).

The design has four pieces, all supplied by dependency injection (dependency injection):

  • QueueOptions holds the capacity and the number of workers, bound with the options pattern, so they can be changed from configuration without changing code (configuration and options).
  • JobQueue is a singleton wrapping the bounded channel. Other code asks for JobQueue in its constructor to enqueue work.
  • WorkerService (BackgroundService) starts N worker loops. Each loop calls WaitToReadAsync and TryRead on the same channel reader (a manual version of await foreach, for a reason explained in stage 4): the channel hands each job to exactly one reader, so the workers share the load automatically, and the number of workers is the limit on concurrency. That limit is deliberate. Ten thousand jobs queued does not mean ten thousand running at once.
  • ResultStore collects the JobResults and can tell you when a given number has arrived.

The demo queues twelve jobs of several kinds, completes the queue, waits for twelve results, stops the host, and then prints the results sorted by id. Completion order across three workers is not deterministic, so we sort; the point is that the set of results is. A small gauge tracks how many jobs ran at the same time, to prove that the limit of three holds.

C# Stage 3: a BackgroundService worker pool with DI and options

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Understand the moving parts:

  • ExecuteAsync returns a Task that represents the whole life of the service. We start three loops, put them in an array and await them together. The service ends when every loop ends, which happens when the channel is completed and drained (normal end) or when the token is cancelled (shutdown).
  • The channel is the load balancer. Three loops read from the same Reader; each job is delivered to exactly one of them. There is no scheduler, no locking in our code, and the worker count is the concurrency limit. Changing Workers in the options changes the throughput and nothing else.
  • Why is the queue a singleton? Producers (an API endpoint) and consumers (the workers) must share one channel. A scoped or transient JobQueue would give everyone their own, and jobs would vanish into queues nobody reads.
  • The delay is registered as a service. Func<TimeSpan, CancellationToken, Task> is a perfectly good dependency. Production registers Task.Delay; tests register a fake. No if (testing) anywhere in the runner.
  • TaskCompletionSource bridges events to await. ResultStore.WaitForAsync(12) returns a task that completes when the twelfth result is added. RunContinuationsAsynchronously is essential: without it, the awaiting code could run inside Add while the lock is held, which can deadlock or surprise (tasks).
  • The gauge uses a compare-and-swap loop to record the maximum without a lock. The result at most 3 is a guarantee of the design; more than one is only true because the jobs take 15 ms each and there are three workers, which is why we print both and treat the first as the real assertion.
  • flaky-3 and flaky-9 succeed on attempt 2, buggy-6 fails once (permanent), down-11 uses all 3 attempts. All twelve results arrive regardless of the order in which they complete.

DisableDefaults = true creates a host with no configuration files, no environment variables and no default logging providers, which is all this demo needs. In a normal worker project you would use Host.CreateApplicationBuilder(args), which adds appsettings.json, environment variables and the console logger, and the numbers in QueueOptions would come from a "Queue" section of the configuration.

Stage 4: timeouts, graceful shutdown and tests

Two production concerns remain.

Timeouts. A job that never finishes occupies a worker forever; enough of them stop the system. Every job needs a time limit. Our runner already has the mechanism: Job.Timeout starts a timer on the per-attempt linked token. When it fires, the job's CancellationToken is cancelled. Note the word cooperative: .NET cannot kill a task. The job must pass its token to the things it awaits (Task.Delay(โ€ฆ, token), HttpClient.GetAsync(url, token), database calls) and stop when it is cancelled. A job that ignores its token cannot be timed out. A timeout counts as a retryable failure in our runner, since the next attempt may be fast.

Graceful shutdown. When the process is told to stop (deploy, Ctrl+C), the host calls StopAsync, which cancels the stoppingToken. We then have a decision to make about jobs in flight. Ours is simple: in-flight jobs receive the cancellation through their token; if they honour it, they end as Cancelled; jobs that are still in the queue are left there (this is where a durable queue would have persisted them). The host waits for ExecuteAsync to finish, up to HostOptions.ShutdownTimeout (30 seconds by default), and then gives up. A shutdown that waits forever for a stuck job is as bad as one that kills everything instantly.

Tests. Time to prove all this. The test suite below uses the same classes, with fake delays where waiting would be slow, and a real host for the shutdown test. It covers:

  1. The retry schedule (a flaky job gets waits of 100 ms and 200 ms).
  2. Permanent errors are not retried; transient errors stop at the attempt limit with the last message.
  3. A job that hangs is timed out, retried, and finally reported as TimedOut.
  4. Cancellation during a job, and during a backoff wait, both give Cancelled and never retry.
  5. Queue behaviour: TryEnqueue refuses when full, EnqueueAsync waits and then completes when a worker frees space, and Complete still delivers everything already queued.
  6. The worker limit holds.
  7. Shutdown with one worker blocked on a stuck job: the job is cancelled and exactly three queued jobs remain.
C# Stage 4: timeouts, graceful shutdown and a test suite

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

A few lines deserve a closer look:

  • Why the "stuck" test is deterministic. With one worker, the worker takes the stuck job first (it is first in the queue) and cannot start any other. The test waits until the job reports that it started, then calls StopAsync. At that moment the three other jobs are certainly still in the channel, so Pending is exactly 3. A test that depended on "probably nothing else started yet" would fail on a busy machine; build the certainty into the test, usually with a TaskCompletionSource signal.
  • StopAsync is the graceful path. It cancels the token, the job's Task.Delay(Infinite, token) throws, the runner turns that into Cancelled, the worker loop sees the cancelled token before taking another job and ends, and ExecuteAsync completes. StopAsync returns after all of that. Nothing was killed.
  • Why not await foreach (... ReadAllAsync(stopping)) in the worker? This is a real trap. ReadAllAsync checks the token only when it has to wait; while items are already buffered it keeps yielding them, so after the shutdown signal the worker went on to take (and run) the three queued jobs. Checking stopping.IsCancellationRequested before each TryRead makes the worker stop taking work the moment shutdown begins. There is still a tiny window between the check and TryRead; a job taken in that window runs with a cancelled token and is reported Cancelled, which is why a durable queue would put such a job back.
  • await body().WaitAsync(TimeSpan.FromSeconds(10)) in the test runner puts a ceiling on every test. A test that hangs because of a bug in the queue is reported as a failure after 10 seconds instead of hanging the whole run.
  • The FakeClock removes waiting from retry tests, the real Task.Delay is used only where the test itself is about time (the 30 ms timeout, the 30 ms wait that shows a writer blocking). Those are short on purpose.
  • Each test builds its own host and disposes it with using. No shared state means no test can influence another.
  • A host does nothing until it is started. Tests that need a full queue simply build the host and do not call StartAsync yet: no worker reads, so the queue fills up. The test that enqueues more jobs than the capacity, on the other hand, must start the host first, or EnqueueAsync would wait for space forever.

Never block on async code inside a job or a worker: .Result, .Wait() and .GetAwaiter().GetResult() hold a thread while waiting. With three workers and three blocked jobs, the pool is stuck, and the thread pool may also run out of threads. Use await all the way down, and pass the CancellationToken to every call that accepts one.

Where to take it next

  1. Durability. Save jobs to a database table (or use a library such as Hangfire, or a broker) so queued jobs survive a restart. The in-memory JobQueue becomes one implementation of an IJobStore interface.
  2. Priorities. Two channels (high and low) and a worker loop that tries the high one first with TryRead, then waits on either. Add a rule that stops low-priority jobs from starving forever.
  3. Scheduled and delayed jobs. "Run in 10 minutes" needs a priority queue ordered by due time and a timer that wakes the workers; .NET's PriorityQueue<TElement, TPriority> is ready for this.
  4. Dead-letter storage. Keep failed jobs (with the exception details) in a list the operators can inspect and re-queue, rather than only a result line.
  5. Metrics. Count queued, running, succeeded, failed and timed-out jobs, expose them as System.Diagnostics.Metrics counters, and graph them (logging).
  6. A web API in front. POST /jobs calls TryEnqueue and answers 202 Accepted or 503 Service Unavailable, using what you learned in the inventory API project.
  7. Circuit breaker. If a dependency keeps failing, stop calling it for a while instead of retrying every job. Libraries such as Polly provide this; writing a small one teaches why retries alone are not enough.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a bounded Channel with FullMode.Wait do when a producer calls WriteAsync and the channel is full?

    WriteAsync completes only when there is room. This makes producers match the speed of the consumers, and it keeps memory use bounded. TryWrite is the variant that returns false instead of waiting.
  2. 2. Why does the retry delay use jitter in addition to exponential backoff?

    Exponential backoff alone makes a crowd of failed jobs retry in lockstep, which can overload the recovering service again. Random jitter spreads the retries out.
  3. 3. A job throws NullReferenceException. What does the runner in this project do, and why?

    Only TransientException (and timeouts) are retried. Other exceptions indicate a defect or bad input that will not fix itself, so the job fails on the first attempt.
  4. 4. Why is the JobQueue registered as a singleton in the DI container?

    The queue is shared state between whoever enqueues and the workers who dequeue. If each consumer of the container got its own JobQueue, jobs would go into queues nobody reads.
  5. 5. What must a job do for its Timeout to be effective?

    Cancellation in .NET is cooperative; the runtime cannot kill a task. The timeout cancels the token, and the job ends only if its awaited calls observe that token.

Summary

  • A bounded Channel<T> is the heart of an in-memory job queue. Capacity gives backpressure: WriteAsync waits, TryWrite refuses. Unbounded queues turn overload into out-of-memory crashes.
  • Complete() ends the channel after delivering what is queued; consumers read with await foreach (โ€ฆ ReadAllAsync(token)) and end by themselves.
  • Separate transient failures (retry) from permanent ones (fail at once). Retry with exponential backoff, a cap and jitter, and make retried jobs idempotent.
  • Treat waiting as a dependency (Func<TimeSpan, CancellationToken, Task>), so tests run instantly and can assert the exact schedule.
  • Run workers in a BackgroundService with DI and options; the number of worker loops reading one channel is the concurrency limit, and the queue must be a singleton.
  • Use a linked token source for per-attempt timeouts, make jobs observe their token (cancellation is cooperative), and give each test a deterministic way to know a job has started (TaskCompletionSource) instead of sleeping and hoping.
  • A graceful shutdown cancels in-flight work, leaves queued jobs untouched (or persists them), and waits for the workers within a time limit.

Next: C# Cheat Sheet