Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 39 ยท Parallel and Scientific Computing

OpenMP Shared-Memory Parallelism in Fortran

Parallelize Fortran loops with OpenMP directives: parallel do, private and shared, reduction, critical, atomic, schedule, collapse, and data race pitfalls.

Modern computers have many cores, and a single Fortran loop uses just one. OpenMP is the standard way to spread work over the cores of one machine. You add directives (special comments starting with !$omp) to your code; a compiler without OpenMP support ignores them as comments, so the program still runs serially. That makes OpenMP a low-risk first step into parallel programming.

The browser cannot run Fortran or OpenMP, so the code below is read-only. The Python stand-ins split work into chunks the way OpenMP does, running them one after the other so the output is deterministic.

Compiling with OpenMP

TEXT
gfortran -O2 -fopenmp sum.f90 -o sum
OMP_NUM_THREADS=4 ./sum

Without -fopenmp the directives are comments. The number of threads comes from the OMP_NUM_THREADS environment variable or from omp_set_num_threads. Use the module omp_lib for the API functions:

FORTRAN
use omp_lib
print *, "max threads:", omp_get_max_threads()

Parallel do

The core directive splits a loop's iterations among threads:

FORTRAN
program saxpy
  use omp_lib
  implicit none
  integer, parameter :: n = 10000000
  real, allocatable :: x(:), y(:)
  real :: a = 2.0
  integer :: i

  allocate(x(n), y(n))
  x = 1.0
  y = 3.0

  !$omp parallel do
  do i = 1, n
    y(i) = a * x(i) + y(i)
  end do
  !$omp end parallel do

  print *, y(1), y(n)       ! 5.0 5.0
end program saxpy

Each thread receives a block of iterations. The loop variable i is automatically private (each thread has its own). Because each iteration touches only y(i), there is no conflict. The !$omp end parallel do line is optional.

Data sharing: shared and private

Inside a parallel region every variable is either shared (all threads see one copy) or private (each thread has its own). Declare it explicitly, and add default(none) so the compiler makes you decide about each variable:

FORTRAN
!$omp parallel do default(none) shared(x, y, a, n) private(i, t)
do i = 1, n
  t = a * x(i)           ! t must be private: every thread uses it
  y(i) = t + y(i)
end do

If t were shared, threads would overwrite each other's scratch value: a data race, the classic bug in parallel code. Its symptom is results that change from run to run.

Reductions

Summing into one variable from many threads is a race too, unless you declare a reduction. Each thread keeps a private partial total and OpenMP combines them at the end:

FORTRAN
real :: total
total = 0.0
!$omp parallel do reduction(+:total)
do i = 1, n
  total = total + x(i) * y(i)
end do
print *, total

Supported operators include +, *, max, min, iand, ior, .and. and .or.. Floating-point addition is not exactly associative, so a parallel sum may differ from the serial one in the last digits: this is rounding, not a bug.

Python Stand-in: chunked reduction like OpenMP

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Scheduling and collapse

How iterations are handed out matters when the work per iteration varies:

Clause Behavior Best for
schedule(static) equal blocks decided in advance uniform work
schedule(dynamic, 100) threads grab chunks of 100 as they finish uneven work
schedule(guided) chunks start large and shrink mixed

For nested loops, collapse(2) merges two loops into one iteration space so there is more to share:

FORTRAN
!$omp parallel do collapse(2) private(i, j)
do j = 1, m
  do i = 1, n
    c(i, j) = a(i, j) + b(i, j)
  end do
end do

Synchronization: critical and atomic

Sometimes threads must update shared data. Use atomic for a single simple update and critical for a larger block:

FORTRAN
!$omp parallel do
do i = 1, n
  if (x(i) > 0.9) then
    !$omp atomic
    hits = hits + 1
  end if
end do

Both slow things down because threads wait for one another, so use them sparingly. Prefer reductions, which scale better.

flowchart LR
    A["serial code"] --> B["!$omp parallel do"]
    B --> C["thread 1<br/>iterations 1..k"]
    B --> D["thread 2<br/>iterations k+1..2k"]
    B --> E["thread N"]
    C --> F["join + combine reductions"]
    D --> F
    E --> F
    F --> G["serial code continues"]

Threads are forked at the directive and joined at its end.

Parallel regions and tasks

A bigger scope avoids creating threads repeatedly:

FORTRAN
!$omp parallel private(tid)
tid = omp_get_thread_num()
!$omp do
do i = 1, n
  ! ...
end do
!$omp end do
!$omp end parallel

Also useful: !$omp single (one thread runs a block), !$omp barrier (everyone waits), and !$omp task for irregular work such as recursion.

Performance reality checks

  • Overhead: forking threads costs microseconds. A loop that runs for a few microseconds is slower in parallel.
  • Memory bandwidth: a loop like y = a*x + y is limited by memory, not arithmetic, so adding cores quickly stops helping.
  • False sharing: threads that write neighboring elements in one cache line fight over it.
  • Amdahl's law: if 10% of the program is serial, the speedup can never exceed 10 times, however many cores you have.

Measure with omp_get_wtime() before and after:

FORTRAN
real(8) :: t0
t0 = omp_get_wtime()
! ... work ...
print *, "seconds:", omp_get_wtime() - t0

Common mistakes

  • A shared scratch variable that should be private.
  • Summing without reduction.
  • A loop-carried dependency inside parallel do (see do concurrent and elemental).
  • Calling a non-thread-safe procedure: those with save variables or module state.
  • Expecting identical floating-point results on every run.
  • Parallelizing tiny loops.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens to !$omp directives when you compile without -fopenmp?

    They are special comments, so the program compiles as ordinary serial code.
  2. 2. Why does a shared scratch variable break a parallel loop?

    Each thread needs its own copy, so declare it private.
  3. 3. How do you safely sum into one variable in a parallel loop?

    A reduction gives each thread a partial sum and combines them at the end.
  4. 4. Which schedule suits iterations of very different cost?

    Dynamic scheduling hands out chunks as threads finish, balancing uneven work.

Summary

  • OpenMP adds parallelism with !$omp comment directives and the -fopenmp flag.
  • parallel do splits loop iterations; make scratch variables private and use default(none).
  • Use reduction for sums and extremes; use atomic or critical rarely.
  • Choose schedule and collapse to balance and expose work; measure with omp_get_wtime.
  • Beware races, overhead, memory bandwidth and Amdahl's law.

Next: Coarrays and parallel images