Running Jupyter Notebooks on the GPU Cluster

Overview

As with other types of computation, by policy users are not allowed to run their jupyter notebooks on the login node (raad2-gfx), but must do so on one of the GPU nodes (gfx1 through gfx4). A jupyter notebook is essentially a web server application that allows the user to interact with it via a standard web browser. Because this application will be running on a GPU node residing on a private network internal to the HPC cluster, certain networking tricks (i.e., "port forwarding") need to be employed to allow a user outside the HPC system to reach the application on the inside. We have tried to make this process a bit friendlier than it normally is by automating it partially.

Before we can get to the point of launching the jupyter notebook though, we first need to install it in our home directory...

Make the Conda Tool Accessible

We can use the conda tool to create a custom environment within our home directory where we will install our own instance of a jupyter notebook. However, before the conda tool can be run, we must source the appropriate file to make it accessible to our bash shell.

source /cm/shared/apps/anaconda/2024.10/etc/profile.d/conda.sh

Alternatively, the line above may be appended to your ~/.bashrc file so you do not have to run this command manually every time you need to use the conda tool. This can be accomplished with:

echo "source /cm/shared/apps/anaconda/2024.10/etc/profile.d/conda.sh" >> ~/.bashrc

After adding this line to the .bashrc file for the first (and only) time, you should log out from your account and then log back in again so that the task of sourcing the conda.sh file is actually performed for your current bash shell.

Install the Jupyter Package

conda create -n myJupyter python=3.11 jupyter

When creating the environment with the command above, we assign it a name with the -n option ("myJupyter_3.8") and we specify a particular version of python we wish to use as a base for this environment with the python=3.8 option. Furthermore, we specify the installation of the jupyter conda package within this base environment.

Launch the Jupyter Notebook on a Raad2 compute Node

The following script may be downloaded to your PC, and saved as jupyter-gfxlauncher.sh.

#!/bin/bash
#
# Jupyter Launcher version 1.0
# Author: itamboli@hbku.edu.qa
# Group: Research Computing Core Group @ HBKU
# Usage: ./jupyter-gfxlauncher.sh <username> [port]
#
#############################################


USER_NAME=$1
#PORT=${2:-8787}

# Define colors
GREEN='\e[32m'
YELLOW='\e[33m'
RED='\e[31m'
RESET='\e[0m'

# Generate a free random port between 50000-60000
find_free_port() {
    while :; do
        port=$((50000 + RANDOM % 10000))
        if ! nc -z localhost $port 2>/dev/null; then
            echo $port
            return
        fi
    done
}

if [ -z "$USER_NAME" ]; then
  printf "\nPlease provide your raad2-gfx username.\n   Usage: $0 <username> \n Example: $0 itambol89\n\n"
  read -p 'Username: ' USER_NAME
  #  echo "Usage: $0 <username> [port]"
  # exit 1
fi

if [ -z "$USER_NAME" ]; then
  echo "Usage: $0 <username> [port]"
  exit 1
fi

PORT=$(find_free_port)
echo "[INFO] Using port $PORT"

echo -e ""
echo -e ">>> Logging into HPC as ${GREEN}$USER_NAME${RESET}"
echo -e ""
echo -e ">>> Will request PBS interactive session, Jupyter on port ${GREEN}$PORT${RESET}"

ssh -t ${USER_NAME}@raad2-gfx.biolab.local "
  echo '>>> When inside compute node, Jupyter will auto-start.'

  # Create helper script inside login node
  mkdir -p ~/.rccg
  cat > ~/.rccg/start_Jupyter_inner.sh <<'INNER'
#!/bin/bash
PORT=$PORT
NODE=\$(hostname)
# Define colors
GREEN='\e[32m'
RESET='\e[0m'
YELLOW='\e[33m'
RED='\e[31m'

echo -e ''
echo -e '>>> Running on compute node:' \${GREEN}\$NODE\${RESET}

# Load environment for modules
if [ -f /etc/profile ]; then
    source /etc/profile
fi


source /cm/shared/apps/anaconda/2024.10/etc/profile.d/conda.sh

conda activate myJupyter

LOGFILE=\$HOME/jupyter_\$\$.log

jupyter lab --no-browser --ip=0.0.0.0 --port=\$PORT > \$LOGFILE 2>&1 &

sleep 5

RAW_URL=\$(grep -m 1 -o 'http://[^ ]*token=[^ ]*' \$LOGFILE)
TOKEN_URL=\$(echo \$RAW_URL | sed 's@http://[^:]*:[0-9]*/@http://localhost:$PORT/@')

#TOKEN_URL=\$(grep -m 1 -o 'http://[^ ]*token=[^ ]*' \$LOGFILE)
echo '=================================================================='
echo -e ''
echo -e '>>> Open another terminal and run following command:'
echo ''
echo -e '    '\${YELLOW}'ssh -L '\$PORT':'\$NODE':'\$PORT' '\$USER'@raad2-gfx.biolab.local'\${RESET}
echo ''
echo -e '>>> Open this URL in your browser:'
echo ''
echo -e '    '\${YELLOW} \$TOKEN_URL \${RESET}
#echo -e '    '\${YELLOW}'http://localhost:'\$PORT\${RESET}
echo ''
echo '=================================================================='

bash
INNER

  chmod +x ~/.rccg/start_Jupyter_inner.sh

  # Launch interactive PBS job and run helper script inside
  qsub -I -N RStudioSession -l select=1:ncpus=4:ngpus=1:mem=16gb -l walltime=01:00:00 -- ~/.rccg/start_Jupyter_inner.sh
"

In order to run this script from a local terminal within your MobaXterm program, do as follows:

 23/09/2025   11:05.14   /home/mobaxterm  ./jupyter-gfxlauncher-pbs.sh itambol89
[INFO] Using port 50161

>>> Logging into HPC as itambol89

>>> Will request PBS interactive session, Jupyter on port 50161
>>> When inside compute node, Jupyter will auto-start.
qsub: waiting for job 163.raad2-cims to start
qsub: job 163.raad2-cims ready


>>> Running on compute node: gfx1
==================================================================

>>> Open another terminal and run following command:

    ssh -L 50161:gfx1:50161 itambol89@raad2-gfx.biolab.local

>>> Open this URL in your browser:

     http://localhost:50161/lab?token=43b1020ece95b1b1464427c50bfa4c5093c31e44bc2dfbcb

==================================================================
[itambol89@gfx1 ~]$

If the following output happens to be present among the output seen above:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:Vhit7kLwb9tE1sexEJ/mW030U1FqDP9Nj/RG8fZrp98.
Please contact your system administrator.
Add correct host key in /ddn/home/fachaud74/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /ddn/home/fachaud74/.ssh/known_hosts:1
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
Port forwarding is disabled to avoid man-in-the-middle attacks.

...then edit your known_hosts file and remove line 1 from it.