Viewing and editing files · Linux Starter Pack

On this page

cat and less
head, tail, wc
Finding lines with grep
Comparing files with diff
Redirection and pipes
nano
Surviving vim
Windows line endings

The examples use two water calculations from the ORCA guide: ~/proj/h2o-sp holds a B3LYP/def2-SVP single point and ~/proj/h2o-opt a geometry optimisation with frequencies.

cat and less

cat prints a whole file to the screen at once. Use it for short files such as inputs.

$ cd ~/proj/h2o-sp
$ cat water.inp
# Single-point energy - water, B3LYP/def2-SVP
! B3LYP def2-SVP
* xyz 0 1
  O   0.000000   0.000000   0.119262
  H   0.000000   0.763239  -0.477047
  H   0.000000  -0.763239  -0.477047
*

Output files are another matter. Even this single point's water.out is 981 lines, and cat scrolls the start past before you can read it. Page through long files with less.

$ less water.out

Key (inside less)	What it does
`Space` / `b`	Forward / back one screen
`↑` `↓`	One line at a time
`g` / `G`	Start / end of file
`/word` + `Enter`	Search forward; `n` for the next match, `N` for the previous
`?word` + `Enter`	Search backward
`q`	Quit

In an ORCA output, G to jump to the end and then ?FINAL SINGLE takes you straight to the last energy. less only reads, so there is no risk of changing the file by accident.

head, tail, wc

head shows the first lines of a file and tail the last ones. -n sets how many (10 by default).

$ tail -n 2 water.out
                             ****ORCA TERMINATED NORMALLY****
TOTAL RUN TIME: 0 days 0 hours 0 minutes 2 seconds 657 msec
$ wc -l water.out ../h2o-opt/water.out
   981 water.out
  3623 ../h2o-opt/water.out
  4604 total

ORCA prints ORCA TERMINATED NORMALLY at the end of a successful run, so tail is the first thing to check when a job finishes. wc -l counts lines. An output much shorter than its siblings has probably died partway.

Watching a running calculation

tail -f shows the end of a file and then keeps waiting, printing new lines as they are written. You can watch which SCF iteration a job has reached. Ctrl+C stops tail; the calculation keeps running.

$ tail -f water.out

Finding lines with grep

grep 'pattern' file prints only the lines that contain the pattern. For finding one number in an output it is much faster than less. Quote patterns that contain spaces.

$ grep 'FINAL SINGLE POINT ENERGY' water.out
FINAL SINGLE POINT ENERGY       -76.321274411145
$ grep 'FINAL SINGLE POINT ENERGY' ../h2o-opt/water.out
FINAL SINGLE POINT ENERGY       -76.426286364586
FINAL SINGLE POINT ENERGY       -76.426353431114
FINAL SINGLE POINT ENERGY       -76.426364583348
FINAL SINGLE POINT ENERGY       -76.426366351564
FINAL SINGLE POINT ENERGY       -76.426369457144

An optimisation prints an energy at every step, so you get several lines. The energy drops a little each time, and the last line is the energy of the optimised geometry (in Hartree).

Option	What it does
`-i`	Ignore case
`-n`	Show line numbers
`-c`	Print only the number of matching lines
`-l`	Print only the names of files that match
`-r`	Search a directory recursively
`-v`	Print lines that do not match

$ grep -n 'TERMINATED NORMALLY' water.out
980:                             ****ORCA TERMINATED NORMALLY****
$ grep -c 'GEOMETRY OPTIMIZATION CYCLE' ../h2o-opt/water.out
4
$ grep -l 'TERMINATED NORMALLY' ~/proj/*/water.out
/home/kim/proj/h2o-opt/water.out
/home/kim/proj/h2o-sp/water.out

The last command lists the calculations that finished normally. With dozens of jobs, that one line is how you find the ones that failed. Regular expressions, and pulling out just the numbers with sed and awk, are covered in the shell guide under grep · sed · awk and Pulling numbers from outputs.

Comparing files with diff

diff shows only the lines that differ between two files. Use it to see how two inputs differ.

$ diff ~/proj/h2o-sp/water.inp ~/proj/h2o-opt/water.inp
1,2c1,2
< # Single-point energy - water, B3LYP/def2-SVP
< ! B3LYP def2-SVP
---
> # Geometry optimization + frequencies + thermochemistry - water
> ! B3LYP D4 def2-TZVP TightOpt Freq

1,2c1,2 means lines 1–2 of the first file were changed into lines 1–2 of the second. < marks lines from the first file, > from the second. The coordinates are identical; only the method and basis differ. No output at all means the files are the same.

Redirection and pipes

By default a command's output goes to the screen. You can send it to a file instead, or feed it into another command. There are two output streams: normal output (stdout) and error messages (stderr).

Syntax	What it does
`cmd > file`	Write output to a file. If the file exists, its contents are erased first.
`cmd >> file`	Append to the end of the file
`cmd 2> file`	Send only error messages to the file
`cmd > file 2>&1`	Send output and errors to the same file
`cmd1 \| cmd2`	Pipe: the output of the first becomes the input of the second

$ grep 'FINAL SINGLE POINT ENERGY' ../h2o-opt/water.out | tail -n 1
FINAL SINGLE POINT ENERGY       -76.426369457144
$ echo '# first line' > notes.txt
$ echo '# second line' >> notes.txt
$ cat notes.txt
# first line
# second line
$ ls water.inp nothing.out > list.txt 2>&1
$ cat list.txt
ls: cannot access 'nothing.out': No such file or directory
water.inp

You use the same syntax to run calculations. ORCA writes its results to the screen (stdout), so you capture them with orca water.inp > water.out.

> overwrites without asking

Running orca water.inp > water.out again wipes the previous water.out, even if that run took three days. Rename the old output or use a fresh directory before rerunning. With set -o noclobber, > refuses to overwrite an existing file.

nano

The easiest editor that runs inside a terminal. Open a file with nano water.inp, move with the arrow keys and type. Shortcuts are listed at the bottom of the screen; ^ means Ctrl.

Key	What it does
`Ctrl+O`, `Enter`	Save
`Ctrl+X`	Exit (asks if there are unsaved changes)
`Ctrl+W`	Search
`Ctrl+K` / `Ctrl+U`	Cut line / paste

Surviving vim

Some clusters have vim but not nano, and commands like git commit may drop you into vim. vim has modes: when it opens, keystrokes are commands, not text. This is the minimum you need.

Key	What it does
`i`	Enter insert mode. When `-- INSERT --` shows at the bottom, you can type text.
`Esc`	Leave insert mode. If something looks wrong, press `Esc` first.
`:wq` + `Enter`	Save and quit
`:q!` + `Enter`	Quit without saving
`/word` + `Enter`	Search (as in `less`)
`u`	Undo

For longer editing on your own machine a graphical editor such as VS Code is more comfortable. Its Remote extensions open files in WSL or on a cluster directly (chapter 10).

Windows line endings

Linux ends each line with a single LF character; Windows uses two, CR+LF. A file written in Windows Notepad and copied to a cluster has an invisible CR (\r) at the end of every line, and scripts then fail with confusing errors.

$ ./run.sh
bash: ./run.sh: /bin/bash^M: bad interpreter: No such file or directory
$ bash run.sh
run.sh: line 2: cd: $'/home/kim/proj/h2o-opt\r': No such file or directory
done

The ^M and \r are the CR. If a directory clearly exists but the shell says it does not, suspect this first. Calculation programs can also misread keywords or file names because of a trailing CR. Check with file and fix with dos2unix.

$ file run.sh water.inp
run.sh:    Bourne-Again shell script, ASCII text executable, with CRLF line terminators
water.inp: ASCII text, with CRLF line terminators
$ cat -A water.inp
! B3LYP def2-SVP^M$
* xyzfile 0 1 water.xyz^M$

# fix it (sudo apt install dos2unix)
$ dos2unix run.sh water.inp
dos2unix: converting file run.sh to Unix format...
dos2unix: converting file water.inp to Unix format...
$ file run.sh water.inp
run.sh:    Bourne-Again shell script, ASCII text executable
water.inp: ASCII text

cat -A makes invisible characters visible: the $ at the end of a line is the LF and the ^M before it is the CR. If the cluster has no dos2unix, sed -i 's/\r$//' run.sh does the same job. In VS Code or Notepad++ you can switch the line ending to LF from the status bar in the bottom-right corner before saving.