Job Queueing and Submission
ALCF's Graphcore POD64 system uses Slurm for job submission and queueing. Below are some of the important commands for using Slurm. For more information refer to Slurm Documentation.
NOTE: Jobs that require IPUs will fail unless launched with
sbatch. NOTE: There is a single Slurm scheduler for the Graphcore POD64.
The Slurm command
srun can be used to run individual Python scripts (or other programs) in parallel with other scripts on a cluster managed by Slurm. An example of
srun usage is shown below. Use the
--ipus= option to specify the number of IPUs required for the run.
Alternatively, these jobs can be submitted to the Slurm workload manager through a batch script by using the
sbatch command. To do this, create a bash script (submit-mnist-poptorch-job.sh here as an example) with the commands that you want to execute.
Then pass the bash script as an input to the
sbatch command as shown below, requesting the number of IPUs required:
squeue command provides information about jobs located in the Slurm scheduling queue.
$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 2572 p64 Graphcor username R 1:12 1 gc-poplar-02
SInfo is used to view partition and node information for a system running Slurm.
For more information, see SInfo.
SCancel is used to signal or cancel jobs, job arrays, or job steps.