Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 39 ยท Parallel and Scientific Computing

Coarrays: Parallel Programming Built into Fortran

Learn Fortran coarrays: images, this_image and num_images, the [ ] cosubscript, sync all, collective subroutines co_sum, critical sections, and compiling with OpenCoarrays.

OpenMP shares memory between threads of one program. Coarrays are Fortran's own language-level model for running many copies of a program, possibly on different machines, that can read and write each other's data. They arrived in Fortran 2008 and gained teams, events and collectives in 2018. Because they are part of the language, you need no separate library API in your source code.

The SPMD model: images

You write one program. When it runs, the system starts several copies, called images, numbered 1 to num_images(). Every image executes the same code, but each has its own memory and its own value of this_image(). This pattern is called SPMD (single program, multiple data).

FORTRAN
program hello_images
  implicit none
  print "(a, i0, a, i0)", "Hello from image ", this_image(), " of ", num_images()
end program hello_images

With four images, four lines print, in no guaranteed order:

TEXT
Hello from image 2 of 4
Hello from image 1 of 4
Hello from image 4 of 4
Hello from image 3 of 4

The order may differ every run, because the images run independently.

Coarray variables

Adding a codimension in square brackets makes a variable visible across images:

FORTRAN
integer :: n[*]              ! one integer per image
real    :: field(100)[*]     ! one array of 100 reals per image

Each image owns its own n. To read or write another image's copy, put the image number in square brackets:

FORTRAN
n = this_image() * 10        ! each writes its own copy
sync all                     ! wait until all images reach this point
if (this_image() == 1) then
  print *, "image 2 has", n[2]
  print *, "image 3 has", n[3]
end if

The [ ] is the cosubscript: it names a different image, while the normal subscript ( ) names an element within one. A reference without [ ] always means the local copy. The cost of a [ ] reference is a communication, so use them deliberately.

Python Stand-in: images as a list of separate memories

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Synchronization

Images run independently, so you must synchronize before reading data another image writes:

Statement Meaning
sync all every image waits until all images arrive (a barrier)
sync images(list) wait only for the listed images
sync memory make previous writes visible
critical ... end critical one image at a time executes the block
error stop stop all images because of an error

Forgetting sync all between a write and a read is the coarray version of a data race: you may read a stale value.

Splitting an array across images

The natural use is domain decomposition: each image holds a slice of a big array.

FORTRAN
program partial_sums
  implicit none
  integer, parameter :: n = 1000
  integer :: me, p, lo, hi, i
  real :: part[*]            ! each image's partial sum
  real :: total

  me = this_image()
  p  = num_images()
  lo = (me - 1) * (n / p) + 1
  hi = me * (n / p)
  if (me == p) hi = n

  part = 0.0
  do i = lo, hi
    part = part + 1.0 / real(i)
  end do

  sync all
  if (me == 1) then
    total = 0.0
    do i = 1, p
      total = total + part[i]
    end do
    print "(a, f10.6)", "Harmonic sum H(1000) = ", total
  end if
end program partial_sums

Each image sums its own block, then image 1 gathers the partial sums. The harmonic number H(1000) is about 7.485471, regardless of how many images run it (apart from rounding).

Python Stand-in: block decomposition and gather

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Collective subroutines

Fortran 2018 adds collectives so you need not write the gather loop. They combine a value over all images:

FORTRAN
call co_sum(part)           ! every image now holds the sum
call co_max(local_max)
call co_min(local_min)
call co_broadcast(config, source_image=1)
call co_reduce(x, operation=my_op)

With co_sum, the example's gather becomes: total = part; call co_sum(total). Collectives are usually faster than hand-written loops because the runtime uses tree algorithms. All images must call the same collective, or the program hangs.

Teams, events and atomics (a first look)

  • Teams split the images into groups with form team and change team, so a sub-problem can use its own numbering.
  • Events (event post, event wait) let one image signal another without a full barrier.
  • Atomic subroutines (atomic_add, atomic_define, atomic_ref) update a remote scalar safely.
  • failed_images() and stat= on synchronization support fault tolerance.

You will need these only for complicated algorithms; many programs use just this_image, num_images, sync all and co_sum.

flowchart TD
    A["start: N images"] --> B["each computes its block"]
    B --> C["sync all"]
    C --> D["co_sum / gather"]
    D --> E["image 1 prints result"]

A typical coarray program: compute locally, synchronize, then combine.

Compiling and running

  • Intel (ifx/ifort): ifx -coarray -O2 prog.f90 && ./a.out (images via FOR_COARRAY_NUM_IMAGES).
  • gfortran with OpenCoarrays: caf prog.f90 -o prog then cafrun -n 4 ./prog.
  • gfortran -fcoarray=single: compiles but runs exactly one image: handy for testing logic.
  • NAG, Cray (HPE), LFortran also support coarrays.

Run on the cluster scheduler as you would any MPI job; OpenCoarrays is built on MPI under the hood.

Coarrays, OpenMP and MPI

Coarrays OpenMP MPI
Memory distributed (also shared) shared distributed
Parallel code part of the language directives library calls
Scale one node to a cluster one node cluster
Best for regular decomposition, readable code loop-level parallelism maximum control and portability

Many large codes use MPI with OpenMP inside each node; coarrays give MPI-like scalability with simpler syntax where compiler support is good.

Common mistakes

  • Reading n[2] before image 2 has written it and before a sync all.
  • Calling sync all on only some images: the program hangs.
  • Printing from every image and expecting ordered output.
  • Dividing work unevenly (remainder rows forgotten).
  • Using [ ] inside the innermost loop; communication dominates run time.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does this_image() return?

    Images are numbered 1 to num_images(); this_image() is your own number.
  2. 2. What does n[3] refer to?

    Square brackets are cosubscripts that select an image; parentheses select array elements.
  3. 3. Why is sync all needed between writing and reading remote data?

    Images run independently, so without synchronization you may read a stale value.
  4. 4. What does call co_sum(x) do?

    co_sum is a collective: all images must call it, and each gets the combined sum (unless result_image is given).

Summary

  • A coarray program runs as several images; this_image() and num_images() identify them.
  • A codimension ([*]) gives each image its own copy; x[k] reads or writes image k's copy.
  • Use sync all (or sync images) before reading remote data; collectives such as co_sum combine values.
  • Teams, events and atomics handle advanced coordination.
  • Compile with -coarray (Intel) or OpenCoarrays caf, and run with several images.

Next: C interoperability