GPyTorch on Polaris

1. Login and queue a job

Login to Polaris

ssh [email protected]

Note

The instructions below should be ran directly from a compute node.

Explicitly, to request an interactive job (from polaris-login):

qsub -A <project> -q debug-scaling -l select=2 -l walltime=01:00:00 -I

Refer to job scheduling and execution for additional information.

2. Load Modules

Load the Anaconda environment module, which contains a PyTorch installation, since GPyTorch has PyTorch as a dependency:

module use /soft/modulefiles
module load conda
conda activate

* Notice that we can check the available modules with "module avail" and check the loaded modules with "module list"

Create a virtual environment with python and activate it

python -m venv --system-site-packages path_to_myenv
source path_to_myenv/bin/activate

Now the bash prompt should show that we're in the environment we just created, and we're good to use pip install

pip install gpytorch==version

Loading environment in future sessions

After the first time, to run the files, simply activate the python_venv on a compute node with

module use /soft/modulefiles
module load conda
source path_to_myenv/bin/activate

3. Using Jupyter Notebook to Run GPyTorch on Polaris

Here is the guide:

Approach 1 - Use ALCF JupyterHub

1. Go to Jupyter Hub of ALCF, click Login Polaris
2. Queue up on a debug node

For the first time only, one needs to set up the environment and Kernel by follow these extra steps

1. Once Jupyter Notebook is launched on a compute node, click "New" and open a terminal
2. Run

   module use /soft/modulefiles
   module load conda
   conda activate
   source <path_to_previously_created_python_venv>/bin/activate
   python -m ipykernel install --user --name python_venv
   

   Note: depending on the system and environment, you might need to install the "ipykernel" package first. The python_venv that I just created has the ipykernel module.

Go back to your .ipynb file, change kernel to python_venv from the dropdown menu, and we'll be good to run GPyTorch!

Approach 2 - Use ssh tunnel

To use ssh tunnel, we first need to be in an interactive session on a compute node. See Part 1, "Log in and queue a job" for more details on this.

On a compute node, follow these steps: 1. On the compute node terminal, do

module use /soft/modulefiles
module load conda
conda activate
jupyter notebook

You should see a line like http://localhost:XXXX/, where XXXX is the port number that jupyter notebook is launched on the compute node, usually the default 8888. If it is not 8888, replace 8888 in the following with your port number.

1. Then, on a new, local terminal, do

   export PORT_NUM=8889
   ssh -L $PORT_NUM:localhost:8888 <[email protected]>
   ssh -L 8888:localhost:8888 your_compute_node
   navigate to localhost:8889 in your browser
   

You should see a jupyter notebook Notice that for the first time doing this, one might need to input some password or weird key. Just follow the direction on that page.

(Essentially, the above steps, using ssh, sets the local port 8889 to listen to the allocated compute node port 8888 which we initiated a jupyter notebook.)

For the first time only, one needs to set up the environment and Kernel by follow these extra steps

Click "New" and open a terminal, and run

module use /soft/modulefiles
module load conda
conda activate
source <path_to_previously_created_python_venv>/bin/activate
python -m ipykernel install --user --name python_venv

Note: depending on the system and environment, you might need to install the "ipykernel" package first. The python_venv that I just created has the ipykernel module.

Go back to your .ipynb file, change kernel to python_venv from the dropdown menu, and we'll be good to run GPytorch!