Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 40 ยท Performance

Threads, Tasks and Distributed Computing

Run Julia code in parallel: Threads.@threads, tasks and channels, atomics and locks to avoid race conditions, Distributed processes and a short GPU overview.

A modern computer has many cores, and a program that uses only one leaves most of the machine idle. Julia has parallelism built in, at three levels: tasks (lightweight units of work that the runtime schedules), threads (several tasks running at the same instant on different cores of one machine), and distributed processes (separate Julia programs, possibly on different computers, that send messages to each other). A fourth route, the GPU, is available through packages.

This lesson introduces each of them, shows the shared-memory bug known as a race condition, and explains how to fix it. Julia's threads cannot run in the browser, so the Julia code is read-only. The Python stand-ins simulate the ideas deterministically: they replay an unlucky interleaving of two workers to show a lost update, and model a channel with a producer and a consumer.

Starting Julia with threads

By default Julia starts with one thread. Ask for more with a command-line flag or an environment variable:

TEXT
julia --threads=4          # or -t 4
julia -t auto              # one thread per CPU core
JULIA_NUM_THREADS=8 julia  # environment variable

Then check how many you have:

JULIA
Threads.nthreads()      # 4
Threads.threadid()      # 1 at the top level: the id of the current thread

In VS Code, set the number of threads in the Julia extension settings. Without extra threads, Threads.@threads still works; it just runs sequentially.

Three levels of parallelism

Tasks Light units of work, scheduled by Julia.
Threads Several cores, one shared memory.
Processes Separate memory, messages between them.
GPU Thousands of tiny cores via packages.
1 / 2

Parallel loops with Threads.@threads

The easiest case is a loop whose iterations are independent: iteration i reads shared input but writes only to its own place in the output.

JULIA
function parallel_squares(n)
    out = Vector{Float64}(undef, n)
    Threads.@threads for i in 1:n
        out[i] = sin(i)^2        # each iteration writes a different slot
    end
    return out
end

@threads splits the range into chunks and hands them to the available threads, and the loop finishes only when all threads are done. The golden rules are: give each iteration its own output slot, do not push to a shared array from several threads, and do not modify shared variables.

A reduction needs one accumulator per chunk, which you combine at the end:

JULIA
function parallel_sum(v)
    chunks = Iterators.partition(v, cld(length(v), Threads.nthreads()))
    tasks = map(chunks) do chunk
        Threads.@spawn sum(chunk)       # one task per chunk
    end
    return sum(fetch.(tasks))           # wait for results, then combine
end

Tasks: Threads.@spawn, fetch and wait

A task wraps a piece of code that the scheduler can run whenever a thread is free. Threads.@spawn expr starts one and returns immediately:

JULIA
t = Threads.@spawn begin
    sleep(1)
    42
end

# ... do other work here ...
fetch(t)       # waits for the task, returns 42; rethrows any error it hit
  • fetch(t) waits and returns the value; wait(t) waits only.
  • @sync begin ... end waits for all @spawn and @async blocks created inside it.
  • @async starts a task on the current thread. It is the right tool for concurrency (overlapping waits, such as network requests), whereas Threads.@spawn gives true parallelism.
  • Tasks are cheap: you can create thousands. They are not operating-system threads; Julia multiplexes them onto the thread pool.
  • Julia 1.9 added thread pools: Threads.@spawn :interactive runs a task on a reserved pool so a UI or server stays responsive while a heavy job runs on the :default pool.

Race conditions

When several threads read and write the same variable, the result depends on timing. This is a data race, and it is a bug even if the program appears to work on your machine.

JULIA
count = 0
Threads.@threads for i in 1:100_000
    global count += 1       # BUG: read, add, write are three separate steps
end
println(count)              # on 4 threads this may print 71384, not 100000

The statement count += 1 is really "read count, add 1, write back". Two threads can both read 5, both compute 6, and both write 6, so one increment is lost. The printed value changes from run to run.

You cannot reproduce a real data race reliably, but you can replay the unlucky ordering on purpose. This stand-in runs two simulated workers step by step, in an interleaved order that loses updates, and then in a protected order:

Python Replaying a lost update

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The interleaved run ends with 1, losing an update. Real threads behave like this unpredictably.

Fixing races: atomics and locks

Julia offers two main tools. An atomic variable supports a few operations that happen as one indivisible step:

JULIA
count = Threads.Atomic{Int}(0)
Threads.@threads for i in 1:100_000
    Threads.atomic_add!(count, 1)
end
println(count[])            # always 100000

A lock makes a larger block of code run by one thread at a time:

JULIA
lk = ReentrantLock()
results = Int[]
Threads.@threads for i in 1:100
    r = i^2                     # the expensive part, done in parallel
    lock(lk) do
        push!(results, r)       # the shared update, done one thread at a time
    end
end

Better still is to avoid sharing. Give each task its own accumulator and combine the results at the end, as in parallel_sum above. Locks and atomics serialize the threads, so if the update is the hot part of the program, they remove much of the speedup. Remember also that many standard containers, such as Array and Dict, are not thread-safe for concurrent writes.

Channels: tasks that talk

A Channel is a queue that tasks can safely put values into and take values out of. It is the Julia tool for a producer and consumer, or a pipeline of stages.

JULIA
ch = Channel{Int}(4)          # holds up to 4 items

producer = Threads.@spawn begin
    for i in 1:10
        put!(ch, i^2)         # blocks if the channel is full
    end
    close(ch)                 # tells consumers there is no more
end

for x in ch                   # iterates until the channel is closed and empty
    println("got ", x)
end

put! waits when the channel is full and take! waits when it is empty. That waiting is how a fast producer is slowed to the pace of its consumer, called backpressure. A Channel can also be created with a function, Channel{Int}(4) do ch ... end, and that function runs as its producer task.

The next stand-in models a bounded channel, using a queue that holds only a few items, to show the producer and consumer alternating:

Python A bounded channel

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You can see the producer stall when two items are waiting, and the consumer stall when none are. Julia's real scheduler does this with real tasks.

Distributed computing

The Distributed standard library starts extra Julia processes. Each has its own memory, so there is no shared-state race, but data must be sent between them.

JULIA
using Distributed
addprocs(4)                          # four workers; or start with julia -p 4

@everywhere function slow_square(x)  # define the function on every process
    sleep(0.1)
    return x^2
end

results = pmap(slow_square, 1:20)    # spread calls over the workers

total = @distributed (+) for i in 1:1_000_000
    i^2                              # partial sums are combined with +
end

Other functions: remotecall(f, worker_id, args...) and fetch for single jobs, and RemoteChannel for channels across processes. The -p and --machine-file flags let you run workers on other computers over SSH, and cluster managers integrate with schedulers like Slurm. Choose processes when the work is large per task, when you need many machines, or when you want isolation. Choose threads for fine-grained work that shares big arrays.

A word on GPUs

Graphics processors run thousands of simple operations at once and are the engine of deep learning. Julia supports them through packages, with a familiar array interface: CUDA.jl for NVIDIA, AMDGPU.jl for AMD, Metal.jl for Apple and oneAPI.jl for Intel. Often only the array type changes:

JULIA
using CUDA
x = CUDA.rand(10_000_000)      # an array in GPU memory
y = x .^ 2 .+ 1                # the broadcast runs on the GPU
sum(y)                         # reduction on the GPU

For custom kernels, KernelAbstractions.jl lets you write one kernel that runs on several vendors' devices. GPUs pay off for big, uniform, arithmetic-heavy workloads, and cost time to copy data to and from the device.

Parallelism rarely gives a perfect speedup. Four threads on a loop rarely run four times faster: memory bandwidth, thread start-up, the garbage collector and serial parts of the program (Amdahl's law) all limit the gain. Always benchmark with @btime using one thread and several.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which command-line option starts Julia with one thread per CPU core?

    -t (or --threads) sets the thread count and accepts auto. -p starts distributed worker processes, which is different.
  2. 2. What is wrong with Threads.@threads for i in 1:n; total += i; end where total is shared?

    Increment is read, add, write. Threads can interleave those steps and overwrite each other's results.
  3. 3. What is the main difference between @async and Threads.@spawn?

    @async gives concurrency on one thread, which suits waiting for I/O. Threads.@spawn can provide real parallelism across cores.
  4. 4. What happens when a producer calls put! on a full Channel?

    Channels block on put! when full and on take! when empty, which keeps producers and consumers in step.
  5. 5. Why do worker processes started with addprocs need @everywhere before a function definition?

    A distributed worker is a separate Julia process with its own memory. Code defined only on the main process is unknown to the workers.

Summary

  • Start Julia with -t auto to use threads; Threads.@threads parallelizes independent loop iterations.
  • Threads.@spawn creates tasks, fetch and wait collect them; @async is for overlapping waits on one thread.
  • Shared mutable state causes races; use atomics, locks, or (best) per-task results combined at the end.
  • Channel connects producers and consumers and applies backpressure.
  • Distributed runs separate worker processes (addprocs, @everywhere, pmap, @distributed), and GPU packages move array work to the graphics card.

Next, write code that writes code in Macros and Metaprogramming.