OpenMP on Aurora
Overview
The OpenMP API is an open standard for parallel programming. The specification document can be found here: https://www.openmp.org. The specification describes directives, runtime routines, and environment variables that allow an application developer to express parallelism (e.g. shared memory multiprocessing and device offloading). Many compiler vendors provide implementations of the OpenMP specification (https://www.openmp.org/specifications).
Setting the environment to use OpenMP on Aurora
The Intel oneAPI Programming Environment is the main environment on Aurora to maximally use the hardware. oneAPI has OpenMP support for both CPU threads and GPU devices. The oneAPI module is loaded by default in your environment:
> module list
Currently Loaded Modules:
1) gcc/11.2.0 3) intel_compute_runtime/release/agama-devel-551 5) libfabric/1.15.2.0 7) cray-libpals/1.3.3
2) mpich/51.2/icc-all-pmix-gpu 4) *oneapi/eng-compiler/2022.12.30.003* 6) cray-pals/1.3.3
However, additional versions of oneAPI with newer compiler versions can be found by adding additional modules to your path:
> module use /soft/modulefiles/
> module avail oneapi
-------------------------------------------------------------------------------- /soft/modulefiles ---------------------------------------------------------------------------------\
oneapi/eng-compiler/2023.05.15.003 oneapi/eng-compiler/2023.10.15.002 oneapi/release/2023.10.15.001 spack-pe-oneapi/0.5-rc1 (D)
oneapi/eng-compiler/2023.05.15.006 oneapi/eng-compiler/2023.12.15.002 (D) oneapi/release/2023.12.15.001 (D)
oneapi/eng-compiler/2023.05.15.007 oneapi/release/2023.05.15.001 spack-pe-oneapi/0.4-rc1
------------------------------------------------------------------------- /opt/aurora/23.073.0/modulefiles -------------------------------------------------------------------------\
oneapi/eng-compiler/2022.12.30.003 (L) oneapi/release/2022.12.30.001
module load oneapi/eng-compiler/2023.10.15.002, for example.
Building on Aurora
The following table shows the compiler and flags
| language | MPI wrapper compiler (underlying compiler) | flag to turn on OpenMP support and target CPU threads | additional flags to target GPU devices |
|---|---|---|---|
| Fortran | mpifort (ifx) | -fiopenmp |
-fopenmp-targets=spir64_gen -Xopenmp-target-backend "-device pvc" |
| C | mpicc (icx) | -fiopenmp |
-fopenmp-targets=spir64_gen -Xopenmp-target-backend "-device pvc" |
| C++ | mpicxx (icpx) | -fiopenmp |
-fopenmp-targets=spir64_gen -Xopenmp-target-backend "-device pvc" |
Running on Aurora
To run, you can run the produced executable or with mpiexec in a job script, and then submit the script to an Aurora queue, like:
$ cat submit.sh
#!/bin/sh
#PBS -l select=1
#PBS -l walltime=0:30:00
#PBS -q EarlyAppAccess
#PBS -A Project
cd ${PBS_O_WORKDIR}
mpiexec -n 1 ./executable
$ # submit to the queue:
$ qsub -l select=1 -l walltime=0:30:00 -q EarlyAppAccess -A Project ./submit.sh
In the above, having the PBS options in the script and on the command line is redundant, but we put it there to show both ways of launching. This submits the script to one node in the EarlyAppAccess queue on Aurora, requesting 30 min. It will charge project Project for the time. You should replace it with your project name.
More details for setting up the job script are in Job Scheduling and Execution section.
Example
$ cat hello.cpp
#include <stdio.h>
#include <omp.h>
int main( int argv, char** argc ) {
printf( "Number of devices: %d\n", omp_get_num_devices() );
#pragma omp target
{
if( !omp_is_initial_device() )
printf( "Hello world from accelerator.\n" );
else
printf( "Hello world from host.\n" );
}
return 0;
}
$ cat hello.F90
program main
use omp_lib
implicit none
integer flag
write(*,*) "Number of devices:", omp_get_num_devices()
!$omp target map(from:flag)
if( .not. omp_is_initial_device() ) then
flag = 1
else
flag = 0
endif
!$omp end target
if( flag == 1 ) then
print *, "Hello world from accelerator"
else
print *, "Hello world from host"
endif
end program main
To compile
$ mpicxx -fiopenmp -fopenmp-targets=spir64_gen -Xopenmp-target-backend "-device pvc" hello.cpp -o c_test
$ mpifort -fiopenmp -fopenmp-targets=spir64_gen -Xopenmp-target-backend "-device pvc" hello.F90 -o f_test