LAMMPS

Overview

LAMMPS is a general-purpose molecular dynamics software package for massively parallel computers. It is written in an exceptionally clean style that makes it one of the more popular codes for users to extend and it currently has dozens of user-developed extensions.

For details bout the code and its usage, see the LAMMPS home page. This page provides information specific to running on Polaris at the ALCF.

Using LAMMPS at ALCF

ALCF provides assistanc with build instructions, compiling executables, submitting jobs, and providing prebuilt binaries (upon request). A collection of Makefiles and submission scripts are available in the ALCF GettingStarted repo here. For questions, contact us at support@alcf.anl.gov.

How to Obtain the Code

LAMMPS is an open-source code, which can be downloaded from the LAMMPS website.

Building on Polaris using KOKKOS package

After LAMMPS has been downloaded and unpacked an ALCF filesystem, users should see a directory whose name is of the form lammps-<version>. One should then see the Makefile lammps-<version>/src/MAKE/MACHINES/Makefile.polaris in recent versions that can be used for compilation on Polaris. A copy of the Makefile is also available in the ALCF GettingStarted repo here. For older versions of LAMMPS, you may need to take an existing Makefile (e.g. Makefile.mpi) for your specific version of LAMMPS used and edit the top portion appropratiately to create a new Makefile.polaris files.

The top portion of Makefile.polaris_nvhpc_kokkos used to build LAMMPS with the KOKKOS package using the NVIDIA compilers is shown as an example.

# polaris_nvhpc_kokkos = Flags for NVIDIA A100, NVIDIA Compiler, MPICH, CUDA
# make polaris_nvhpc_kokkos -j 16

SHELL = /bin/sh

# ---------------------------------------------------------------------
# compiler/linker settings
# specify flags and libraries needed for your compiler

KOKKOS_DEVICES = Cuda,OpenMP
KOKKOS_ARCH = Ampere80
KOKKOS_ABSOLUTE_PATH = $(shell cd $(KOKKOS_PATH); pwd)
KOKKOS_CUDA_OPTIONS = "enable_lambda,disable_malloc_async"
export NVCC_WRAPPER_DEFAULT_COMPILER = nvc++

CRAY_INC = $(shell CC --cray-print-opts=cflags)
CRAY_LIB = $(shell CC --cray-print-opts=libs)

#$(info CRAY_INC = ${CRAY_INC})
#$(info CRAY_LIB = ${CRAY_LIB})

CC =        $(KOKKOS_ABSOLUTE_PATH)/bin/nvcc_wrapper
CCFLAGS =  -g -O3 -mp -DLAMMPS_MEMALIGN=64
CCFLAGS += $(CRAY_INC)
SHFLAGS =   -fPIC
DEPFLAGS =  -M

LINK =      $(CC)
LIB =
LIB += $(CRAY_LIB)
SIZE =      size

With the appropriate LAMMPS Makefile in place an executable can be compiled as in the following example, which uses the NVIDIA compilers.

cd lammps-<version>/src
make yes-KOKKOS
make polaris_nvhpc_kokkos -j 16

Running Jobs on Polaris

An example submission script for running a KOKKOS-enabled LAMMPS executable is below as an example. Additional information on LAMMPS application flags and options is described on the LAMMPS website.

#!/bin/sh
#PBS -l select=64:system=polaris
#PBS -l place=scatter
#PBS -l walltime=0:15:00
#PBS -l filesystems=home:grand:eagle
#PBS -q prod
#PBS -A Catalyst

export MPICH_GPU_SUPPORT_ENABLED=1

NNODES=`wc -l < $PBS_NODEFILE`

# per-node settings
NRANKS=4
NRANKSSOCKET=2
NDEPTH=8
NTHREADS=1
NGPUS=4

NTOTRANKS=$(( NNODES * NRANKS ))

EXE=/home/knight/bin/lmp_polaris_nvhpc_kokkos
EXE_ARG="-in in.reaxc.hns -k on g ${NGPUS} -sf kk -pk kokkos neigh half neigh/qeq full newton on "

# OMP settings mostly to quiet Kokkos messages

MPI_ARG="-n ${NTOTRANKS} --ppn ${NRANKS} --depth=${NDEPTH} --cpu-bind depth --env OMP_NUM_THREADS=${NTHREADS} --env OMP_PROC_BIND=spread --env OMP_PLACES=cores"

COMMAND="mpiexec ${MPI_ARG} ${EXE} ${EXE_ARG}"
echo "COMMAND= ${COMMAND}"
${COMMAND}

Performance Notes

Some useful information on accelerator packages and expectations can be found on the LAMMPS website here.