Containers on Polaris
Since Polaris is using NVIDIA A100 GPUs, there can be portability advantages with other NVIDIA-based systems if your workloads use containers. In this document, we'll outline some information about containers on Polaris including how to build custom containers, how to run containers at scale, and common gotchas.
Container creation can be achieved one of two ways either by using Docker on your local machine as mentioned in Docker section of Theta(KNL) and publishing it to DockerHub, or by using a Singularity recipe file and building on a Polaris worker node. If you are not interested in building a container and only want to use the available containers, you can read the section on available containers.
Singularity
The container system on Polaris is singularity
. You can set up singularity with a module (this is different than, for example, ThetaGPU!):
# To see what versions of singularity are available:
module avail singularity
# To load the Default version:
module load singularity
# To load a specific version:
module load singularity/3.8.7 # the default at the time of writing these docs.
Which Singularity?
There used to be a single singularity
tool, which in 2021 split after some turmoil. There are now two singularity
s: one developed by Sylabs, and the other as part of the Linux Foundation. Both are open source, and the split happened around version 3.10. The version on Polaris is from Sylabs but for completeness, here is the Linux Foundation's version. Note that the Linux Foundation version is renamed to apptainer
- different name, roughly the same thing though divergence may happen after 2021's split.
Build from Docker Images or Argonne Github container registry
Docker containers require root privileges, which users do not have on Polaris. That doesn't mean all your docker containers aren't useful, though. If you have an existing docker container, you can convert it to singularity pretty easily on the login node. To build the latest NVIDIA container for PyTorch you can run the following:
module load singularity
singularity build pytorch:22.06-py3.sing docker://nvcr.io/nvidia/pytorch:22.06-py3
Note that latest
here mean when these docs were written, summer 2022. It may be useful to get a newer container if you need the latest features. You can find the PyTorch container site here. The tensorflow containers are here (though note that LCF doesn't prebuild the TF-1 containers typically). You can search the full container registry here.
You can also use our custom built containers using Github OCI container registry. Here's a list of containers distributed by ALCF staff tailored for Polaris.
Build with a Recipe
You can also build a singularity container using a recipe file. Detailed instructions for recipe construction are available on the Singularity Recipe Page. You can also check our singularity recipe example for building a mpich version 4 container on Polaris.
Once you have a recipe file, you can build it on Polaris, but only on compute nodes. You can launch an interactive job using the attribute singularity_fakeroot=true
to build on a compute node.
qsub -I -A <project_name> -q <queue> -l select=1 -l walltime=60:00 -l singularity_fakeroot=true -l filesystems=home:eagle:grand
You need to replace the <project_name>
with the appropriate project to charge and <queue>
with debug
, or preemptable
queues since we only request a single node.
After your interactive job has started, you need to load the singularity
module on the compute node and export the proxy variables for internet access. Then you can build the container as shown below.
module load singularity
export HTTP_PROXY=http://proxy.alcf.anl.gov:3128
export HTTPS_PROXY=http://proxy.alcf.anl.gov:3128
export http_proxy=http://proxy.alcf.anl.gov:3128
export https_proxy=http://proxy.alcf.anl.gov:3128
singularity build --fakeroot <image_name>.sif <def_filename>.def
Alternatively, you can just pull the mpich 4 image distributed by us and build on top of it
Running Singularity container on Polaris
Example submission script on Polaris
To run a container on Polaris you can use the submission script described here. Below we have described the submission script for your understanding.
First we define our job and our script takes the container name as an input parameter.
#!/bin/sh
#PBS -l select=2:system=polaris
#PBS -q debug
#PBS -l place=scatter
#PBS -l walltime=0:30:00
#PBS -l filesystems=home:grand
#PBS -A <project_name>
cd ${PBS_O_WORKDIR}
echo $CONTAINER
We move to current working directory and enable network access at run time by setting the proxy. We also load singularity.
# SET proxy for internet access
module load singularity
export HTTP_PROXY=http://proxy.alcf.anl.gov:3128
export HTTPS_PROXY=http://proxy.alcf.anl.gov:3128
export http_proxy=http://proxy.alcf.anl.gov:3128
export https_proxy=http://proxy.alcf.anl.gov:3128
This is important for system (Polaris - Cray) mpich to bind to containers mpich. Set the following environment variables
ADDITIONAL_PATH=/opt/cray/pe/pals/1.1.7/lib/
module load cray-mpich-abi
export SINGULARITYENV_LD_LIBRARY_PATH="$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH:$ADDITIONAL_PATH"
Set the number of ranks per node spread as per your scaling requirements
# MPI example w/ 16 MPI ranks per node spread evenly across cores
NODES=`wc -l < $PBS_NODEFILE`
PPN=16
PROCS=$((NODES * PPN))
echo "NUM_OF_NODES= ${NODES} TOTAL_NUM_RANKS= ${PROCS} RANKS_PER_NODE= ${PPN}"
Finally launch your script
echo C++ MPI
mpiexec -hostfile $PBS_NODEFILE -n $PROCS -ppn $PPN singularity exec -B /opt -B /var/run/palsd/ $CONTAINER /usr/source/mpi_hello_world
echo Python MPI
mpiexec -hostfile $PBS_NODEFILE -n $PROCS -ppn $PPN singularity exec -B /opt -B /var/run/palsd/ $CONTAINER python3 /usr/source/mpi_hello_world.py
The job can be submitted using:
Available containers
If you just want to know what containers are available, here you go.
- For running mpich/MPI containers on Polaris, it can be found here
- For running databases on Polaris. It can be found here
- For using shpc - that allows for running containers as modules. It can be found here
- Some containers are found in /soft/containers
The latest containers are updated periodically. If you have trouble using containers, or request a newer or a different container please contact ALCF support at support@alcf.anl.gov
.
Troubleshooting
-
Permission Denied Error: One may get a
permission denied
error during the build process, due to a nasty permission setting, quota limitations, or simply due to an unresolved symbolic link. You can try one of the solutions below:- Check your quota and delete any unnecessary files.
- Clean-up singularity cache,
~/.singularity/cache
, and set the singularity tmp and cache directories as below: - Make sure you are not on a directory accessed with a symlink, i.e. check if
pwd
andpwd -P
returns the same path. - If any of the above doesn't work, try running the build in your home directory.
-
Mapping to rank 0 on all nodes: This is mainly due to container mpich not binding to system mpich. It is imperative for the container to have mpich which can bind dynamically to system mpich at runtime. Ensure your submission script has the following variables and modules loaded (see below). If this does not resolve, ensure the containers mpich is built with the '--disable-wrapper-rpath' flag. Please refer to this link to find examples of building a mpich based container from scratch and running on Polaris.
ADDITIONAL_PATH=/opt/cray/pe/pals/1.1.7/lib/
module load cray-mpich-abi
export SINGULARITYENV_LD_LIBRARY_PATH="$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH:$ADDITIONAL_PATH"
singularity exec -B /opt -B /var/run/palsd/
-
libmpi.so.40 not found: This may be due to mpich binding to the wrong system mpich. Try removing .conda & .cache & .local folders from your home directory. Also rebuild your container and try again.
-
Containers built with openmpi may not work correctly. Please ensure your container is built with mpich and the base image is of Debian architecture (For e.g. Ubuntu) image.