Shell scripting
less repetition with bash

Running the same calculation on fifty molecules, then pulling one energy out of each of fifty outputs into a table. By hand that takes an afternoon; a twenty-line script does it in seconds. This guide is about writing those twenty lines.

Shell bash 4+ · Examples ORCA · Quantum ESPRESSO · LAMMPS outputs · Schedulers PBS Pro · Slurm

About this guide

A shell script is the commands you would type in a terminal, saved in a file. The syntax is a little odd and has a few traps, but the part you need for computational work is small: variables and loops, three text tools (grep, sed, awk), and the format a job scheduler expects.

If the terminal itself is new to you, start with the Linux starter pack. This guide assumes you know commands like cd, ls and cp, and how file permissions work.

Chapters

01 · BASICS
Your first script
Shebang, execute permission, variables and quoting, command substitution.
02 · CONTROL
Conditions and loops
if and test, for and while, case.
03 · FUNCTIONS
Functions and arguments
$1 and $@, getopts, functions and return values.
04 · ROBUST
Writing safer scripts
set -euo pipefail, exit codes, trap, ShellCheck.
05 · TEXT
grep · sed · awk
Search, replace, cut by column, chain with pipes.
06 · EXTRACT
Pulling numbers from outputs
ORCA energies and frequencies, QE total energy, LAMMPS thermo tables.
07 · BATCH
PBS · Slurm job scripts
Requesting resources, setting up the environment, using scratch.
08 · ARRAYS
Job arrays and parameter sweeps
Submitting many molecules or many parameter values at once.
09 · CHEAT SHEET
Cheat sheet
Syntax, text tools and scheduler commands on one page.

Conventions

In code boxes, a line starting with $ is a command typed at the terminal, and the rest is output. Boxes without $ show the contents of a script file.

bash only

Other shells exist, such as zsh (the macOS default) and fish, but cluster login shells and job scripts are almost always bash. With #!/bin/bash as the first line, a script runs under bash whatever your interactive shell is.