grep · sed · awk · Shell Scripting Guide

On this page

Example files
grep: finding lines
Regular expression basics
sed: editing lines
awk: working with columns
sort · uniq · cut · paste · tr · column · wc
Practice on xyz files

Example files

The examples use two xyz files and one ORCA output. opt.out is the water geometry optimisation + frequency output (3623 lines) from the ORCA guide.

$ cat methanol.xyz
6
methanol
C     -0.046520    0.662749    0.000000
O     -0.046520   -0.754989    0.000000
H     -1.086345    0.976632    0.000000
H      0.437275    1.070835    0.889981
H      0.437275    1.070835   -0.889981
H      0.861339   -1.051121    0.000000

In an xyz file the first line is the atom count, the second is a comment, and every line after that holds an element symbol and x, y, z coordinates in Å. water.xyz has the same layout with three atoms.

grep: finding lines

grep pattern file prints every line that contains the pattern.

$ grep -c '^H' methanol.xyz
4
$ grep -n '^O' water.xyz methanol.xyz
water.xyz:3:O      0.000000    0.000000    0.119262
methanol.xyz:4:O     -0.046520   -0.754989    0.000000
$ grep -v '^H' methanol.xyz
6
methanol
C     -0.046520    0.662749    0.000000
O     -0.046520   -0.754989    0.000000

You only need a few options.

Option	Effect
`-i`	Ignore case
`-n`	Show line numbers
`-c`	Print only the number of matching lines
`-v`	Print lines that do not match
`-l`	Print only the names of files that contain a match
`-r`	Search every file below a directory
`-A N`, `-B N`	Also print N lines after / before each match
`-w`	Match whole words only
`-E`	Extended regex (`+`, `\|`, `()` work without backslashes)
`-F`	Treat the pattern as a fixed string, not a regex

-r -l tells you which of many job directories finished. -A and -B are handy when you want a table header together with the lines around it.

$ grep -r -l 'TERMINATED NORMALLY' ../orca
../orca/05-casscf-h2/h2_casscf.out
../orca/04-solvation-smd/water.out
../orca/03-dlpno-ccsdt/water.out
../orca/02-opt-freq/water.out
../orca/01-single-point/water.out
$ grep -B1 -A1 'THE OPTIMIZATION HAS CONVERGED' opt.out
                    ***********************HURRAY********************
                    ***        THE OPTIMIZATION HAS CONVERGED     ***
                    *************************************************
$ grep -c 'FINAL SINGLE POINT ENERGY' opt.out
5

The 5 is four optimisation cycles plus the final energy evaluation after convergence. -w only matches when the pattern is bounded by non-word characters. TightOpt in the input line below contains Opt but not as a whole word, so the second command prints nothing.

$ grep 'Opt' water.inp
! B3LYP D4 def2-TZVP TightOpt Freq
$ grep -w 'Opt' water.inp

Regular expression basics

grep, sed and awk read patterns as regular expressions. For output files this much is enough.

Symbol	Meaning	Example
`.`	Any single character	`H.` matches `H1`, `He`, `H`
`*`	Previous character, zero or more times	`^H.*-`: starts with H and has a `-` later
`+`	Previous character, one or more times (`-E`)	`[0-9]+`: one or more digits
`[...]`	One of the listed characters	`[CHO]`, `[0-9.]`
`^`, `$`	Start and end of line	`^H`, `Eh$`
`\s`	Whitespace (GNU extension)	`^\s+`: leading indentation
`a\|b`	a or b (`-E`)	`^(C\|O)`

$ grep -E '^(C|O) ' methanol.xyz
C     -0.046520    0.662749    0.000000
O     -0.046520   -0.754989    0.000000
$ grep -E '^\s+[0-9]+:\s+[0-9.]+ cm' opt.out
     0:       0.00 cm**-1
     1:       0.00 cm**-1
     2:       0.00 cm**-1
     3:       0.00 cm**-1
     4:       0.00 cm**-1
     5:       0.00 cm**-1
     6:    1615.68 cm**-1
     7:    3780.78 cm**-1
     8:    3885.42 cm**-1

If the text you are looking for contains *, . or [, add -F. Without it, the * in cm**-1 is read as a repetition and unrelated lines match.

$ grep 'cm**-1' opt.out | head -3
Rotational constants in cm-1:            27.545905            14.281345             9.405174 
Rotational constants in cm-1:    27.545929    14.281357     9.405182 
Chem.Eur.J. 2012 18 9955 using a reference frequency of 100.0 cm-1
$ grep -F 'cm**-1' opt.out | head -3
     0:       0.00 cm**-1
     1:       0.00 cm**-1
     2:       0.00 cm**-1

Single-quote your patterns

Inside single quotes the shell leaves * and $ alone. Use double quotes only when the pattern has to include a shell variable.

sed: editing lines

sed reads a file line by line, applies its commands and prints the result. The original file is not touched. The command you will use most is substitution, s/find/replace/.

$ sed 's/0/X/' water.xyz
3
water
O      X.000000    0.000000    0.119262
H      X.000000    0.763239   -0.477047
H      X.000000   -0.763239   -0.477047
$ sed 's/0/X/g' water.xyz
3
water
O      X.XXXXXX    X.XXXXXX    X.119262
H      X.XXXXXX    X.763239   -X.477X47
H      X.XXXXXX   -X.763239   -X.477X47

Without g only the first match on each line changes. If the replacement contains /, pick a different delimiter so you don't have to escape it.

$ sed 's|water|water, B3LYP-D4/def2-TZVP|' water.xyz
3
water, B3LYP-D4/def2-TZVP
O      0.000000    0.000000    0.119262
H      0.000000    0.763239   -0.477047
H      0.000000   -0.763239   -0.477047

-n turns off automatic printing and p prints the addressed lines, so together they select lines. An address can be a line number (3,5), the last line ($), a regex (/^H/) or a range between two regexes (/start/,/end/). d deletes the addressed lines.

$ sed -n '3,5p' methanol.xyz
C     -0.046520    0.662749    0.000000
O     -0.046520   -0.754989    0.000000
H     -1.086345    0.976632    0.000000
$ sed '/^H/d' methanol.xyz
6
methanol
C     -0.046520    0.662749    0.000000
O     -0.046520   -0.754989    0.000000
$ sed -n '/VIBRATIONAL FREQUENCIES/,/NORMAL MODES/p' opt.out
VIBRATIONAL FREQUENCIES
-----------------------

Scaling factor for frequencies =  1.000000000  (already applied!)

     0:       0.00 cm**-1
     1:       0.00 cm**-1
     2:       0.00 cm**-1
     3:       0.00 cm**-1
     4:       0.00 cm**-1
     5:       0.00 cm**-1
     6:    1615.68 cm**-1
     7:    3780.78 cm**-1
     8:    3885.42 cm**-1


------------
NORMAL MODES

Editing files in place: -i

With -i sed writes the result back to the file instead of the screen. There is no undo, so get used to -i.bak, which keeps a backup with that suffix.

$ sed -i.bak '2s/.*/renamed/' w2.xyz
$ ls w2.xyz*
w2.xyz
w2.xyz.bak
$ head -2 w2.xyz
3
renamed

sed -i differs on macOS

GNU sed on Linux edits without a backup with sed -i 's/a/b/' f. BSD sed on macOS always takes a suffix after -i, so the no-backup form there is sed -i '' 's/a/b/' f. sed -i.bak behaves the same on both, which makes it the safe choice in scripts.

awk: working with columns

awk splits each line on whitespace into $1, $2, …; $0 is the whole line, NF the number of fields on that line, $NF the last field, and NR the current line number. A program is a list of pattern {action} pairs, and each action runs only on lines that match its pattern.

$ awk 'NR > 2 {print NR, NF, $1, $NF}' methanol.xyz
3 4 C 0.000000
4 4 O 0.000000
5 4 H 0.000000
6 4 H 0.889981
7 4 H -0.889981
8 4 H 0.000000
$ awk '/^H/ {print $2}' methanol.xyz
-1.086345
0.437275
0.437275
0.861339
$ awk '$1 == "H" && $4 > 0' methanol.xyz
H      0.437275    1.070835    0.889981

With no action, awk prints the whole line. Be careful with header lines when the pattern is a comparison: $4 < 0 without NR > 2 also matches the first two lines, which have no fourth field.

$ awk '$4 < 0' methanol.xyz
6
methanol
H      0.437275    1.070835   -0.889981

For other separators use -F: -F, for CSV, -F: for colons. Pass shell variables in with -v name=value.

BEGIN, END, sums and averages

A BEGIN block runs once before the first line, an END block once after the last. Variables start at 0 without being declared, so collecting a sum and a count and dividing in END gives an average.

$ awk 'NR > 2 {z += $4; n++} END {print z, n, z/n}' water.xyz
-0.834832 3 -0.278277
$ awk 'NR > 2 {z += $4; n++} END {printf "mean z = %.4f over %d atoms\n", z/n, n}' water.xyz
mean z = -0.2783 over 3 atoms

printf takes C-style format strings: %d integer, %.4f four decimals, %-8s string left-aligned in 8 columns, %12.6f right-aligned in 12. You add the newline \n yourself.

$ awk 'BEGIN {print "element  x"} NR > 2 {printf "%-8s %8.3f\n", $1, $2}' methanol.xyz
element  x
C          -0.047
O          -0.047
H          -1.086
H           0.437
H           0.437
H           0.861

Use awk for floating-point arithmetic

bash arithmetic $(( )) is integer-only. Division truncates, and a decimal point is a syntax error.

$ echo $(( 3 / 2 ))
1
$ echo $(( -76.426 * 627.5 ))
bash: line 1: -76.426 * 627.5 : arithmetic syntax error: invalid arithmetic operator (error token is ".426 * 627.5 ")
$ awk 'BEGIN {print -76.426369457144 * 627.5095}'
-47958.3
$ awk 'BEGIN {printf "%.2f\n", -76.426369457144 * 627.5095}'
-47958.27

awk does all arithmetic in double precision, but print shows only six significant digits. For energies, where the digits matter, set the precision with printf.

sort · uniq · cut · paste · tr · column · wc

Small tools that sit between grep, sed and awk in a pipeline.

Command	What it does
`sort`	Sort lines. `-n` numeric, `-g` numeric including exponents, `-r` reverse, `-k4,4` by the fourth column
`uniq`	Collapse adjacent duplicate lines; `-c` adds counts. Usually follows `sort`
`cut`	Cut columns. `-d, -f2` second comma-separated field, `-c1-2` characters 1–2
`paste`	Join files side by side; `-d,` sets the separator
`tr`	Translate characters: `tr 'A-Z' 'a-z'`, `tr ' ' '\n'`
`column -t`	Align columns for reading; `-s,` sets the input separator
`wc -l`	Count lines

$ tail -n +3 methanol.xyz | awk '{print $1}' | sort | uniq -c
      1 C
      4 H
      1 O
$ tail -n +3 methanol.xyz | sort -k4,4g
H      0.437275    1.070835   -0.889981
C     -0.046520    0.662749    0.000000
H      0.861339   -1.051121    0.000000
H     -1.086345    0.976632    0.000000
O     -0.046520   -0.754989    0.000000
H      0.437275    1.070835    0.889981
$ paste -d, el.txt z.txt
O,0.119262
H,-0.477047
H,-0.477047
$ printf 'name,E\nwater,-76.3213\nmethanol,-115.5\n' | column -t -s,
name      E
water     -76.3213
methanol  -115.5

tail -n +3 prints from line 3 to the end, a quick way to skip the two xyz header lines. uniq only merges duplicates that are next to each other, so without sort in front the counts are wrong.

Practice on xyz files

Atoms per element

The sort | uniq -c above can also be done in a single awk call, since awk arrays take strings as keys. The order of for (e in n) is unspecified, so pipe through sort if order matters.

$ awk 'NR > 2 {n[$1]++} END {for (e in n) print e, n[e]}' methanol.xyz | sort
C 1
H 4
O 1

Shifting coordinates

Move the water molecule 1.5 Å along z. The two header lines are printed unchanged and next skips to the next line.

$ awk -v dz=1.5 'NR <= 2 {print; next} {printf "%-2s %12.6f %12.6f %12.6f\n", $1, $2, $3, $4 + dz}' water.xyz
3
water
O      0.000000     0.000000     1.619262
H      0.000000     0.763239     1.022953
H      0.000000    -0.763239     1.022953

To move the geometric centre to the origin you need the average before you can print anything, so read the file twice. NR == FNR is true only while reading the first file (FNR restarts at 1 for each file).

$ awk 'NR == FNR {if (FNR > 2) {x += $2; y += $3; z += $4; n++}; next}
       FNR <= 2 {print; next}
       {printf "%-2s %12.6f %12.6f %12.6f\n", $1, $2 - x/n, $3 - y/n, $4 - z/n}' methanol.xyz methanol.xyz
6
methanol
C     -0.139271     0.333592     0.000000
O     -0.139271    -1.084146     0.000000
H     -1.179096     0.647475     0.000000
H      0.344524     0.741678     0.889981
H      0.344524     0.741678    -0.889981
H      0.768588    -1.380278     0.000000

Converting to other formats

Turn the file into an ORCA coordinate block (* xyz charge multiplicity … *), assuming a neutral singlet.

$ awk 'NR == 1 {print "* xyz 0 1"} NR > 2 {printf "  %-2s %12.6f %12.6f %12.6f\n", $1, $2, $3, $4} END {print "*"}' methanol.xyz
* xyz 0 1
  C     -0.046520     0.662749     0.000000
  O     -0.046520    -0.754989     0.000000
  H     -1.086345     0.976632     0.000000
  H      0.437275     1.070835     0.889981
  H      0.437275     1.070835    -0.889981
  H      0.861339    -1.051121     0.000000
*

For a spreadsheet or Python, number the atoms and join the fields with commas.

$ awk 'BEGIN {print "idx,el,x,y,z"} NR > 2 {print NR - 2 "," $1 "," $2 "," $3 "," $4}' water.xyz
idx,el,x,y,z
1,O,0.000000,0.000000,0.119262
2,H,0.000000,0.763239,-0.477047
3,H,0.000000,-0.763239,-0.477047

In awk, values written next to each other are concatenated; print a, b with a comma puts a space between them. Pulling energies and geometries out of real output files is the subject of the next chapter.