Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 37 ยท State and Events

Queue, Concurrency and Rate Limits

Tune Gradio's queue: default_concurrency_limit, per-event limits, max_size, batching, status updates and rate limiting. Learn how many users an app can serve and why.

Every Gradio event goes through a queue. The queue is what lets ten people use a slow model at the same time without crashing it: requests wait in line, a fixed number run at once, and everyone sees their position. Left at its defaults, the queue works for small demos; on a busy app or a heavy model you must tune it. This lesson explains the knobs and how to reason about capacity. Gradio code is read-only; stand-ins let you do the arithmetic and simulate a batch.

The mental model

flowchart LR
    U1["User 1"] --> Q["Queue<br/>(max_size)"]
    U2["User 2"] --> Q
    U3["User 3"] --> Q
    Q --> W["Workers<br/>(concurrency_limit)"]
    W --> F["Your function"]

Requests wait in the queue; only concurrency_limit run at a time.

  • Concurrency limit: how many runs of a given event may be active at once. The default is 1 per event in classic versions (so a second user waits for the first to finish), and configurable globally.
  • Queue size (max_size): how many waiting requests are allowed. When full, new requests are refused with a "queue is full" message instead of waiting forever.
  • Status updates: the page shows "In queue: position 3 of 5" and an estimated wait.

Configuring the queue

Python
demo = gr.Blocks()
demo.queue(
    default_concurrency_limit=2,   # runs allowed at once for every event
    max_size=20,                   # at most 20 waiting requests
)
demo.launch(max_threads=40)

Per-event limits override the default:

Python
btn.click(run_model, inp, out, concurrency_limit=1)      # heavy GPU model
btn.click(fast_lookup, inp, out2, concurrency_limit=10)  # cheap function

Events can share a pool with concurrency_id: all events with the same id together obey one limit, which is the right choice when two buttons call the same GPU model:

Python
a.click(model_fn, x, y, concurrency_id="gpu", concurrency_limit=1)
b.click(other_model_fn, x, z, concurrency_id="gpu")

A limit of None removes it: use that only for functions that are cheap and thread-safe. How to choose: set the limit to the number of simultaneous runs your hardware handles well. One GPU model that fills memory gets 1. A function that waits on a remote API (and uses no CPU) can have a high limit, because the time is spent waiting.

max_threads in launch() is the size of the thread pool for synchronous functions. A concurrency limit larger than the thread count cannot be reached. Async functions (async def) do not use the thread pool and scale further for I/O-bound work.

Batching

Some models are much faster on a batch than on single inputs. Gradio can collect several waiting requests and call your function once with lists:

Python
def classify_batch(texts):
    labels = model.predict(texts)          # one call for many users
    return [labels]                        # a list per output component

gr.Interface(classify_batch, "text", "text", batch=True, max_batch_size=16)

With batch=True your function receives a list per input and returns a list per output, with the same length and order. Gradio waits briefly to fill the batch, runs it and hands each user their own item. Typical speed-ups apply to neural networks on GPUs; for simple Python code batching only adds complexity.

Rate limits and fairness

The queue is not a rate limiter. A user can still submit repeatedly. Options:

  • gr.Request gives the client's IP (request.client.host) and session hash. Keep a small dictionary of recent timestamps per IP and raise gr.Error when too frequent (behind a proxy, read x-forwarded-for and make sure the proxy is trusted).
  • Put a reverse proxy such as nginx or a CDN in front for real limits.
  • trigger_mode="once" stops a single browser from stacking identical runs.
  • Use auth or logins for expensive models so you know who is calling.

Reading the numbers

Capacity is simple arithmetic. If a call takes T seconds and the limit is C, the app serves about C / T requests per second. For T = 4 s and C = 2, that is 0.5 requests per second, or 30 per minute. A user who arrives when 10 requests are waiting waits roughly 10 * T / C = 20 seconds. If that is too long, add capacity (more workers, a faster model, caching) or limit the queue size so people are told to retry rather than waiting forever.

Stand-in 1: queue capacity maths

Python Stand-in: wait time and throughput

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The numbers show why doubling the concurrency halves the wait only if the hardware can really run two jobs in parallel.

Stand-in 2: batching and a per-user rate limiter

Python Stand-in: batch function and rate limiter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Alice is refused at the third call because two happened in the last 10 seconds, and allowed again at second 11. The module-level dictionary is fine here because rate limiting is exactly the kind of shared state that should be global.

A tuning recipe

  1. Measure how long one call takes on your real hardware, with realistic inputs.
  2. Decide how many can run together without exhausting memory (GPU, RAM).
  3. Set concurrency_limit and max_size to match, and use concurrency_id for shared models.
  4. Cache or batch if throughput is still too low.
  5. Load test with several browser tabs or a script using the Python client.
  6. Add rate limits and authentication before exposing the app widely.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does concurrency_limit control?

    It limits simultaneous runs of an event (or a shared concurrency_id).
  2. 2. What happens when max_size is reached?

    A bounded queue protects the app and tells users to retry.
  3. 3. With batch=True, what does your function receive?

    Batching collects several requests into one call with equal-length lists.
  4. 4. A job takes 4 s and concurrency is 2. Roughly how many jobs can finish per minute?

    Two at a time at 4 s each is 0.5 jobs per second, or 30 per minute.

Summary

  • Every event goes through a queue; concurrency_limit sets simultaneous runs and max_size the waiting room.
  • Use concurrency_id to share a limit between events that use one resource.
  • Batching serves many requests with one model call.
  • The queue is not a rate limiter; add per-user limits and authentication for costly models.
  • Capacity is concurrency divided by seconds per job; measure before tuning.

Next: Wrapping a Model in a Gradio Demo