GPyTorch on Polaris¶

Login to Polaris

ssh [email protected]

Note

The instructions below should be run directly from a compute node.

Explicitly, to request an interactive job (from polaris-login):

qsub -A <project> -q debug-scaling -l select=2 -l walltime=01:00:00 -I

Refer to job scheduling and execution for additional information.

2. Load Modules¶

Load the Anaconda environment module, which contains a PyTorch installation, since GPyTorch has PyTorch as a dependency:

module use /soft/modulefiles
module load conda
conda activate

* Notice that we can check the available modules with "module avail" and check the loaded modules with "module list."

Create a virtual environment with Python and activate it:

python -m venv --system-site-packages path_to_myenv
source path_to_myenv/bin/activate

Now the bash prompt should show that we're in the environment we just created, and we're good to use pip install:

pip install gpytorch==version

Loading environment in future sessions¶

After the first time, to run the files, simply activate the Python virtual environment on a compute node with:

module use /soft/modulefiles
module load conda
source path_to_myenv/bin/activate

3. Using Jupyter Notebook to Run GPyTorch on Polaris¶

Here is the guide:

Approach 1 - Use ALCF JupyterHub¶

Go to Jupyter Hub of ALCF, click Login Polaris.
Queue up on a debug node.

For the first time only, one needs to set up the environment and kernel by following these extra steps:

Once Jupyter Notebook is launched on a compute node, click "New" and open a terminal.

Run:

module use /soft/modulefiles
module load conda
conda activate
source <path_to_previously_created_python_venv>/bin/activate
python -m ipykernel install --user --name python_venv

Note: Depending on the system and environment, you might need to install the "ipykernel" package first. The python_venv that I just created has the ipykernel module.

Go back to your .ipynb file, change the kernel to python_venv from the dropdown menu, and we'll be good to run GPyTorch!

Approach 2 - Use SSH Tunnel¶

To use an SSH tunnel, we first need to be in an interactive session on a compute node. See Part 1, "Login and queue a job" for more details on this.

On a compute node, follow these steps: 1. On the compute node terminal, do:

module use /soft/modulefiles
module load conda
conda activate
jupyter notebook

You should see a line like http://localhost:XXXX/, where XXXX is the port number that Jupyter Notebook is launched on the compute node, usually the default 8888. If it is not 8888, replace 8888 in the following with your port number.

Then, on a new, local terminal, do:

export PORT_NUM=8889
ssh -L $PORT_NUM:localhost:8888 <[email protected]>
ssh -L 8888:localhost:8888 your_compute_node
navigate to localhost:8889 in your browser

You should see a Jupyter Notebook. Notice that for the first time doing this, one might need to input some password or key. Just follow the directions on that page.

(Essentially, the above steps, using SSH, set the local port 8889 to listen to the allocated compute node port 8888 where we initiated a Jupyter Notebook.)

For the first time only, one needs to set up the environment and kernel by following these extra steps:

Click "New" and open a terminal, and run:

module use /soft/modulefiles
module load conda
conda activate
source <path_to_previously_created_python_venv>/bin/activate
python -m ipykernel install --user --name python_venv