NWChem
NWChem is a parallel quantum chemistry mainly written in Fortran77 and that uses MPI and OpenMP for distributed and multicore computing. NWChem was designed to solve large-scale electronic structure calculations with Hartree-Fock, Density Functional Theory, and other wavefunction correlated methods. See full features at http://www.nwchem-sw.org.
Using NWChem at ALCF
ALCF provides binaries and compiling instructions for NWChem. For questions, contact us at support@alcf.anl.gov.
How to Obtain the Code
NWChem is an open-source code the official web site is http://www.nwchem-sw.org. Change log and release announcements could be found here http://www.nwchem-sw.org/index.php/Download.
Official NWChem source could be downloaded from the project’s github.com repository https://github.com/nwchemgit/nwchem/releases. For detailed compiling options, please visit the wiki page of NWchem, https://github.com/nwchemgit/nwchem/wiki/Compiling-NWChem. The follow instructions were tested in for the version 6.8.1.
cat nwchem_envs_681.sh
export NWCHEM_TOP=/projects/nwchem/nwchem-6.8.1
export NWCHEM_TARGET=LINUX64
export USE_MPI=y
export USE_CPPRESERVE=y
export NWCHEM_MODULES="all"
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export USE_OPENMP=1
export USE_KNL=1
export USE_OPENMP=y
export INTEL_64ALIGN=1
export USE_NOIO=1
export USE_GAGITHUB=1
export CRAYPE_LINK_TYPE=dynamic
export export ARMCI_NETWORK=MPI_TS
export BLAS_SIZE=4
export BLASOPT=" -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lpthread -lm -ldl"
export SCALAPACK=" -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lmkl_blacs_intelmpi_lp64 -lpthread -liomp5 -ldl"
export BLAS_LIB=$BLASOPT
export LAPACK_LIB=$BLASOPT
export SCALAPACK_SIZE=4
export USE_KNL=1
export USE_64TO32=1
cd $NWCHEM_TOP/src
make nwchem_config
make 64_to_32
make
Alternatively, binaries can be found in the folders under /soft/applications/nwchem
Running Jobs on Theta
The following script “submit.sh” is an example to run Nwchem in Theta a job of 8 nodes with 64 MPI ranks per node. The job can be submitted with command "qsub submit.sh"
cat submit.sh
#!/bin/bash
#COBALT -n 8
#COBALT -t 30
#COBALT -A myproject
module load atp
bin=/soft/applications/nwchem/6.8/bin/nwchem
echo "Running Cobalt Job $COBALT_JOBID."
rpn=64
df -k
export MPICH_GNI_MAX_EAGER_MSG_SIZE=16384
export MPICH_GNI_MAX_VSHORT_MSG_SIZE=10000
export MPICH_GNI_MAX_EAGER_MSG_SIZE=131072
export MPICH_GNI_NUM_BUFS=300
export MPICH_GNI_NDREG_MAXSIZE=16777216
export MPICH_GNI_MBOX_PLACEMENT=nic
export MPICH_GNI_LMT_PATH=disabled
export COMEX_MAX_NB_OUTSTANDING=6
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin
aprun -n $((COBALT_JOBSIZE*rpn)) -cc depth -d1 -j1 $bin test.nw
qsub submit.sh