Conditions and loops

Exit status and $?

Every command leaves behind an integer between 0 and 255 when it finishes: its exit status. 0 means success, anything else means failure. Every conditional in the shell is driven by this number. The exit status of the previous command is in $?.

$ grep -q "ORCA TERMINATED NORMALLY" water_sp.out
$ echo $?
0
$ grep -q "ORCA TERMINATED NORMALLY" methane_opt.out
$ echo $?
1
$ grep -q "ORCA TERMINATED NORMALLY" nofile.out
grep: nofile.out: No such file or directory
$ echo $?
2

grep returns 0 if it finds a match, 1 if it does not, and 2 on an error such as a missing file. -q suppresses all output.

$? is overwritten by every command you run, and echo is a command too. If you need the value later, copy it into a variable straight away.

$ ls nofile
ls: cannot access 'nofile': No such file or directory
$ echo "exit status: $?"
exit status: 2
$ echo "now: $?"
now: 0

if, elif, else

if is followed by a command. If that command's exit status is 0, the then block runs. Unlike languages that put a boolean expression in parentheses, any command can serve as the condition.

if grep -q "ORCA TERMINATED NORMALLY" water.out; then
    echo "finished"
elif grep -q "ERROR" water.out; then
    echo "error"
else
    echo "still running or killed"
fi

To put then on the same line you need a ; before it. The block is closed by fi. Indentation has no meaning to bash but makes the code readable.

Tests with [[ ]]

Checks such as "does this file exist", "is this string empty" or "is this number larger" are written with [[ ]]. The spaces after [[ and before ]] are required.

Do not compare numbers with < and >. Inside [[ ]], > compares strings in dictionary order, so 10 comes out smaller than 9. Use -gt or the arithmetic form (( )) for integers.

$ [[ 10 > 9 ]] && echo yes || echo no
no
$ [[ 10 -gt 9 ]] && echo yes || echo no
yes
$ n=12; (( n > 8 )) && echo big
big
$ f=conf03.xyz; [[ $f == *.xyz ]] && echo "xyz file"
xyz file

The [ ] in older scripts is another name for the test command. If the variable is not quoted it breaks as soon as the value is empty. [[ ]] is bash syntax: no word splitting happens inside it and pattern matching works. Use [[ ]] in bash scripts.

$ x=
$ [ $x = "" ] && echo empty
bash: [: =: unary operator expected
$ [[ $x == "" ]] && echo empty
empty

&& and ||

A && B runs B only if A succeeded. A || B runs B only if A failed. Short conditions are often written this way instead of with if.

mkdir -p opt && cp water.xyz opt/
[[ -f water.xyz ]] || echo "water.xyz is missing"
cd "$workdir" || exit 1

A && B || C looks like if-else but is not the same: C also runs when B fails. Only use it when B is something that cannot fail, like echo.

for loops

for var in list assigns each item in turn to the variable and runs the block. For files, put a wildcard where the list goes.

for f in *.xyz; do
    mol=${f%.xyz}
    mkdir -p "$mol"
    cp "$f" "$mol/"
done

You will often see for f in $(ls *.xyz). It is wrong: the output of ls is split at spaces, so a file with a space in its name becomes two items. A wildcard is never split.

$ for f in $(ls *.xyz); do echo "[$f]"; done
[ethanol]
[conf1.xyz]
[methane.xyz]
[water.xyz]
$ for f in *.xyz; do echo "[$f]"; done
[ethanol conf1.xyz]
[methane.xyz]
[water.xyz]

If nothing matches, bash leaves the pattern as literal text, so the loop runs once with a "file" called *.gjf. With shopt -s nullglob the pattern becomes an empty list and the loop does not run at all.

$ for f in *.gjf; do echo "[$f]"; done
[*.gjf]
$ shopt -s nullglob
$ for f in *.gjf; do echo "[$f]"; done
$ shopt -u nullglob

Brace ranges produce lists of numbers. A leading zero as in {01..10} pads them to the same width. Variables do not work inside the braces: {1..$n} is not expanded, so use seq or a C-style loop instead.

$ echo {1..5} {01..10..3} conf{a,b}
1 2 3 4 5 01 04 07 10 confa confb
$ n=3; echo {1..$n}
{1..$n}
$ for (( i=0; i<5; i++ )); do
>     r=$(awk -v i="$i" 'BEGIN { printf "%.2f", 0.90 + 0.05*i }')
>     echo "scan_$i: R = $r"
> done
scan_0: R = 0.90
scan_1: R = 0.95
scan_2: R = 1.00
scan_3: R = 1.05
scan_4: R = 1.10

The last example generates bond lengths for a scan. The loop counts in integers and awk computes the decimal values. The > at the start of the lines is the prompt bash shows while a command continues.

Reading a file line by line with while read

Suppose mols.txt lists molecule names, charges and spin multiplicities.

water 0 1
methane 0 1
nitrate -1 1

Feeding the file to while read -r reads it one line at a time. With several variable names, the line is split at whitespace and whatever is left over goes into the last variable.

$ while read -r name charge mult; do
>     echo "$name: charge $charge, mult $mult"
> done < mols.txt
water: charge 0, mult 1
methane: charge 0, mult 1
nitrate: charge -1, mult 1

Writing the same thing as cat mols.txt | while read ... has a catch: variables changed inside the loop do not survive it, because each part of a pipeline runs in its own subshell.

$ count=0
$ cat mols.txt | while read -r line; do count=$((count + 1)); done
$ echo "count = $count"
count = 0
$ count=0
$ while read -r line; do count=$((count + 1)); done < mols.txt
$ echo "count = $count"
count = 3
$ count=0
$ while read -r f; do count=$((count + 1)); done < <(find jobs -name '*.out')
$ echo "count = $count"
count = 3

Redirect a file with < file and another command's output with process substitution, < <(command), and the loop runs in the current shell. Mind the space between the two <.

case

To compare one value against several patterns, case reads better than a chain of ifs. Here it dispatches on file extension.

for f in water.xyz run.inp job.out notes.txt geom.gjf; do
    case "$f" in
        *.xyz)       echo "$f: coordinates" ;;
        *.inp|*.gjf) echo "$f: input" ;;
        *.out|*.log) echo "$f: output" ;;
        *)           echo "$f: skipped" ;;
    esac
done

water.xyz: coordinates
run.inp: input
job.out: output
notes.txt: skipped
geom.gjf: input

Each branch ends with ;; and only the first matching branch runs. *) catches everything else. Inside any loop, continue skips to the next item and break leaves the loop.

Example: did the calculations finish normally?

Say jobs/ holds one directory per calculation, each with an .out file named after the directory. When ORCA finishes normally it writes ****ORCA TERMINATED NORMALLY**** at the end of the output. The script sorts jobs by whether that line is there.

#!/bin/bash
# check each job directory under jobs/ for a normal ORCA termination

ok=0
bad=0

for d in jobs/*/; do
    d=${d%/}
    name=${d##*/}
    out="$d/$name.out"

    if [[ ! -f $out ]]; then
        echo "NOOUT  $name"
        bad=$((bad + 1))
    elif grep -q "ORCA TERMINATED NORMALLY" "$out"; then
        echo "OK     $name"
        ok=$((ok + 1))
    else
        echo "FAIL   $name"
        bad=$((bad + 1))
    fi
done

echo "normal: $ok, not normal: $bad"

$ ./check_jobs.sh
NOOUT  benzene_opt
FAIL   methane_opt
OK     water_opt
OK     water_sp
normal: 2, not normal: 2

The trailing / in jobs/*/ selects directories only, and ${d%/} removes it again. A FAIL may simply be a job that is still running; check the queue with qstat or squeue. Pulling the energies out as well and putting them in a table is the subject of chapter 06.