CHAPTER 18 · ADVANCED · APPLICATIONS

Parallel execution and HPC

The -nk/-nd/-ni parallelization levels, scaling intuition for small and large systems, and HPC habits.

On this page

The parallelization cheat sheet

mpirun -np 32 pw.x -nk 8 -in input.in > output.out
Flag Alias Meaning Guidance
-nk -npool k-point pools Most efficient. Make it divide the k-point count, and leave nproc/nk large enough for the FFT distribution
-nb -nband Band groups Useful for hybrids and EXX
-nt -ntg FFT task groups Many bands and many cores
-nd -ndiag Diagonalization group Must be a square number. Large systems only
-ni -nimage Image parallelism NEB, phonons, hp.x

The default split (no flags) distributes plane waves (G-vectors). Small cells with many k-points want -nk; large cells with few k-points lean on the G-vector split and -nd. If you run out of memory, lower -nk: every pool holds its own copy of the charge density.

Measured: OpenMP threads on small systems

The two-atom Si SCF from this guide, on a 16-core WSL machine with the conda-forge build:

Setting Wall time
OMP_NUM_THREADS=16 (threads only) about 2 minutes
OMP_NUM_THREADS=1 + mpirun -np 6 pw.x -nk 6 2.8 seconds

Conda builds ship with OpenMP enabled, and on small systems the idle-thread spinning can cost you an order of magnitude or two. On a local learning machine, set OMP_NUM_THREADS=1 explicitly and parallelize with MPI plus -nk. On HPC, if you use hybrid MPI×OpenMP, keep ranks-per-node × threads = physical cores.

Read the timing breakdown first

Base your parallelization choices on the timing block at the end of the output.

     init_run     :    ...
     electrons    :    ...      ← the whole SCF
     c_bands      :    ...      ← diagonalization (if large: -nd, diagonalization)
     sum_band     :    ...
     fft + ffts   :    ...      ← FFT (if large: G-vector split, -nt)

In PWSCF : ... CPU ... WALL, CPU much larger than WALL points at thread spin; WALL much larger than CPU points at I/O or communication.

HPC habits

  • Always set max_seconds. The run saves state and exits cleanly just before the queue limit; continue with restart_mode='restart'.
  • Point outdir at scratch (the fast parallel filesystem). Writing wavefunctions into your home directory slows you and everyone else down.
  • Record the QE version, pseudopotential paths and names, and the commit of your scripts inside the batch script, so you can later tell which settings produced which data.
  • For large scans (convergence tests, U scans, MD frame recomputation), workflow tools like AiiDA and ASE beat shell loops on reproducibility.

A typical PBS/Slurm line:

mpirun -np $SLURM_NTASKS pw.x -nk 8 -nd 4 -in feo.scf.in > feo.scf.out
Common mistakes

npool must divide nproc: your -nk does not divide the total rank count. some processors have no planes: more ranks than FFT z-planes (too many ranks for a small cell). Parallel setup errors are collected in section 4 of the R3 error dictionary.

  • Every example page's "Run" section uses commands chosen by this chapter's rules. For splitting hp.x over q-points, see Example E12.