PBS · Slurm job scripts

On a cluster you don't run calculations on the login node. You hand a job script to the scheduler, which runs it on a compute node once resources are free. Logging in and the basic concepts are in the HPC chapter of the Linux starter pack. This chapter is about the script itself.

A PBS Pro job script

A job script is an ordinary bash script with #PBS lines at the top. bash sees them as comments; qsub reads them as resource requests. They must come before the first command.

#!/bin/bash
#PBS -N water-opt
#PBS -l select=1:ncpus=8:mem=16gb
#PBS -l walltime=12:00:00
#PBS -j oe
#PBS -o water-opt.log

cd "$PBS_O_WORKDIR" || exit 1
echo "running on $(hostname) with $NCPUS cores"

The job starts in your home directory, so the first command is cd "$PBS_O_WORKDIR", the directory where you ran qsub. PBS Pro tells you the number of cores you got in $NCPUS. Queue names, per-node core and memory limits and the maximum walltime vary between clusters; check your cluster's documentation or qstat -Q.

The same job in Slurm

Slurm uses #SBATCH lines. The option names differ, but the request is the same.

#!/bin/bash
#SBATCH --job-name=water-opt
#SBATCH --nodes=1
#SBATCH --ntasks=8
#SBATCH --cpus-per-task=1
#SBATCH --mem=16G
#SBATCH --time=12:00:00
#SBATCH --output=water-opt.%j.log

cd "$SLURM_SUBMIT_DIR" || exit 1
echo "running on $(hostname) with $SLURM_NTASKS tasks"

--ntasks is the number of MPI processes and --cpus-per-task the number of threads per process. MPI programs such as ORCA want --ntasks=8 --cpus-per-task=1; OpenMP-only programs such as xtb want --ntasks=1 --cpus-per-task=8.

module and conda

Job scripts run in a non-interactive shell, and non-interactive bash does not read ~/.bashrc. The conda activate and PATH settings that work in your terminal are missing inside the job, so set up the environment in the script.

# software provided by the cluster
module purge
module load openmpi/4.1.6

# conda environment: source conda.sh before activating
source "$HOME/miniforge3/etc/profile.d/conda.sh"
conda activate xtb

# record what was picked up
which xtb mpirun
module list

Calling conda activate without sourcing conda.sh fails with an error saying the shell is not initialised. Find module names and versions with module avail. If a job reports module: command not found, your cluster needs its module init file sourced first; the path is in the cluster documentation.

Running ORCA in parallel

Three things matter when ORCA runs on several cores.

• Call orca by its full path. ORCA locates its helper programs relative to its own path in parallel steps, and a bare orca makes parallel runs fail.
• Make %pal nprocs in the input match the cores you requested. More processes than cores slows everything down; fewer leaves cores idle.
• Load the OpenMPI version your ORCA build expects. The version is listed on the ORCA download page and in the release notes.

! B3LYP def2-SVP Opt
%pal nprocs 8 end
%maxcore 1500
* xyz 0 1
  O   0.000000   0.000000   0.119262
  H   0.000000   0.763239  -0.477047
  H   0.000000  -0.763239  -0.477047
*

%maxcore is memory per core in MB. ORCA sometimes goes over it, so keep nprocs × maxcore at about 75% of the memory you request. Here 8 × 1500 MB = 12 GB against a 16 GB request.

#!/bin/bash
#PBS -N water-opt
#PBS -l select=1:ncpus=8:mpiprocs=8:mem=16gb
#PBS -l walltime=12:00:00
#PBS -j oe

cd "$PBS_O_WORKDIR" || exit 1

module load openmpi/4.1.6
export PATH="$HOME/opt/orca_6_1_0:$PATH"
export LD_LIBRARY_PATH="$HOME/opt/orca_6_1_0:${LD_LIBRARY_PATH:-}"
ORCA=$(command -v orca)

"$ORCA" water.inp > water.out 2>&1

mpiprocs=8 tells PBS Pro to reserve slots for 8 MPI processes. The ORCA install path and OpenMPI version are examples; change them to match your setup.

Node-local scratch

ORCA writes many large temporary files during a run. On a shared file system (home or project directories) that is slow and loads the file server for everyone. Copy the input to the compute node's local disk, run there, and bring back only the results. PBS Pro often creates a per-job temporary directory and puts it in $TMPDIR. On Slurm clusters the location varies, so look up the scratch path in your cluster's documentation.

JOB=water
SCR=$(mktemp -d "${TMPDIR:-/tmp}/orca.XXXXXX")
cp "$JOB.inp" "$SCR"/
sed -i "s/^%pal.*/%pal nprocs $NCPUS end/" "$SCR/$JOB.inp"

copy_back() {
    cp "$SCR"/*.xyz "$SCR"/*.gbw "$SCR"/*.hess "$PBS_O_WORKDIR"/ 2>/dev/null
    rm -rf "$SCR"
}
trap copy_back EXIT

cd "$SCR" || exit 1
"$ORCA" "$JOB.inp" > "$PBS_O_WORKDIR/$JOB.out" 2>&1

trap copy_back EXIT runs copy_back however the script ends: normally, after an ORCA error, or on a termination signal from the scheduler. The output file goes straight to the submit directory, so you can follow it with tail -f while the job runs and it survives if the job dies. The sed line rewrites %pal to the number of cores the job actually received. Traps and exit codes are covered in chapter 04.

OpenMP thread count

OpenMP programs (xtb, CREST, the BLAS behind numpy) start OMP_NUM_THREADS threads. If it is unset they may try to use every core on the node and collide with other jobs. Set it to the cores you received.

export OMP_NUM_THREADS="${NCPUS:-${SLURM_CPUS_PER_TASK:-1}}"

Submitting, monitoring, cancelling

Job states are Q (queued), R (running) and F (finished, with -x) in PBS, and PD (pending) and R (running) in Slurm. Slurm's sacct shows how much time and memory a finished job actually used.

sacct -j 123456 --format=JobID,JobName,State,Elapsed,MaxRSS,ExitCode

Job dependencies

To start a frequency calculation only after the optimisation succeeds, give the second job a dependency. afterok means "start only if the first job exits with status 0". If the first job fails, the second stays queued and you have to delete it yourself.

# PBS Pro: keep the job ID that qsub prints
OPT=$(qsub opt.pbs)
qsub -W depend=afterok:"$OPT" freq.pbs

# Slurm: --parsable prints only the job ID
OPT=$(sbatch --parsable opt.sh)
sbatch --dependency=afterok:"$OPT" freq.sh

Common failures

A script written in a Windows editor has \r at the end of every line, and bash reads it as part of the command.

$ bash job.sh
job.sh: line 2: cd: $'run1\r': No such file or directory
done
$ file job.sh
job.sh: Bourne-Again shell script, ASCII text executable, with CRLF line terminators
$ sed -i 's/\r$//' job.sh

Strip the \r as in the last line, or run dos2unix job.sh. Setting your editor to LF line endings keeps it from coming back.