First steps on HPC

Login nodes and compute nodes

A cluster is tens to thousands of computers linked by a fast network. Each computer is a node. When you ssh in, you land on a login node: a shared entry point where everyone organises files, writes inputs, submits jobs and does short compiles. The calculations themselves run on compute nodes, which you normally do not ssh into directly.

Who gets which compute nodes, and when, is decided by the scheduler. You submit a job script stating the resources you need ("8 cores, 16 GB of memory, 2 hours") and the commands to run. The scheduler puts the job in a queue and starts it on a compute node when the resources are free. The two common schedulers are PBS Pro (used at NUS HPC, among others) and Slurm. The ideas are the same; the command names differ.

Loading software with module

A cluster has several versions of ORCA, Quantum ESPRESSO, LAMMPS, compilers and MPI libraries installed. Putting all of them on your PATH would cause clashes, so you load only what you need with the module command. module load edits environment variables such as PATH for you (chapter 05).

$ module avail              # everything available
$ module avail orca         # only modules with orca in the name
$ module load orca/6.0.1    # load one (names and versions differ per cluster)
$ module list               # what is loaded now
$ module unload orca/6.0.1  # unload one
$ module purge              # unload everything

Module names and versions differ between clusters, so do not copy the name above; check with module avail. Leaving out the version loads the default, and the default can change when the admins update it. For calculations that go into a paper, pin the version.

conda in your home directory

Tools that are not available as modules (xtb, CREST, ASE, pymatgen and so on) can be installed with conda. You have no admin rights on a cluster, so install miniforge into your home directory; the steps are in the xtb guide.

$ conda create -n chem -c conda-forge python=3.12 ase xtb
$ conda activate chem

A conda environment can take several GB. If your home quota is small, keep the number of environments down, remove unused ones with conda env remove -n NAME, and clear the package cache with conda clean --all. Mixing a module-loaded MPI with a conda environment easily leads to library clashes, so in a job script activate only what that job needs.

Storage

Cluster storage is usually split by purpose. Names, paths, sizes and deletion policies differ per cluster, so read your site's documentation.

Go over your quota and nothing more can be written, so a running job dies in the middle of writing its output. Each cluster has its own quota command; to see what is taking up space use du, and for free space on a file system use df.

$ du -sh ~/proj/*           # size of each directory
$ df -h ~                   # size and free space of the file system holding home

Submitting a first job

A minimal PBS Pro job script. Lines beginning with #PBS are comments to bash but resource requests to the scheduler. Save it as run.pbs next to h2o.inp in the job directory ~/proj/h2o-opt/.

#!/bin/bash
#PBS -N h2o-opt
#PBS -l select=1:ncpus=8:mem=16gb
#PBS -l walltime=02:00:00
#PBS -j oe

cd $PBS_O_WORKDIR
module load orca/6.0.1
$(which orca) h2o.inp > h2o.out

-N is the job name; select=1:ncpus=8:mem=16gb asks for one node with 8 cores and 16 GB; walltime is the time limit, after which the job is killed whether or not it has finished. -j oe merges standard output and error into one file. The job starts in your home directory, so cd $PBS_O_WORKDIR moves back to where you submitted it. ORCA must be called by its full path for parallel runs, hence $(which orca), and %pal nprocs 8 end in the input must match ncpus (see the ORCA guide).

$ cd ~/proj/h2o-opt
$ qsub run.pbs              # submit; prints the job ID
$ qstat -u $USER            # status of your jobs
$ qstat -f <jobid>          # full details of one job
$ qdel <jobid>              # cancel

The S column of qstat is the state: Q queued, R running, H held. When the job ends, a file h2o-opt.o<jobid> appears in the submission directory with messages from the scheduler and any errors from the script. If h2o.out is empty, open that file first; a mistyped module name or running past the walltime are the usual suspects.

On Slurm

Same ideas. The common commands and directives side by side:

Using scratch and submitting many jobs at once with job arrays are covered in the shell scripting guide chapters on job scripts and job arrays.

Interactive jobs

Sometimes you want to type commands on a compute node yourself: to check that a program runs, or for post-processing too heavy for the login node. An interactive job gives you a shell on a compute node once resources are allocated.

# PBS Pro
$ qsub -I -l select=1:ncpus=4:mem=8gb -l walltime=01:00:00

# Slurm
$ srun --ntasks=1 --cpus-per-task=4 --mem=8G --time=01:00:00 --pty bash

When the hostname in the prompt changes to a compute node, you are in. exit releases the resources. If the queue is busy, the shell may take a while to appear.

Etiquette

• Do not run calculations on the login node. Anything that keeps a core busy for more than a few seconds goes in a job.
• Request what the program actually uses. Asking for 32 cores for a calculation that uses 8 lengthens everyone's wait, including yours, since the job waits until 32 are free.
• Give walltime some margin, but do not ask for the maximum out of habit. Shorter jobs often slot into gaps sooner.
• Do not submit thousands of tiny jobs separately. Use a job array or run several calculations inside one job.
• Do not poll the scheduler constantly, as in watch -n 1 qstat.
• Clean up scratch files from finished calculations.