File transfer

Every command in this chapter runs on your laptop. Remote paths are written host:path, and the host is the vanda alias from chapter 06. Without an alias, spell it out as e0123456@vanda.nus.edu.sg:path.

scp

scp is cp across machines: scp source destination, with a host on one side.

# upload: laptop → cluster
$ scp h2o.inp h2o.xyz vanda:~/proj/h2o-opt/

# download into the current directory (.)
$ scp vanda:~/proj/h2o-opt/h2o.out .

# a whole directory
$ scp -r vanda:~/proj/h2o-opt .

In a remote path, ~ is your home directory on the cluster. Forget the colon and scp happily copies to a local file called vanda. For a handful of files, once, scp is all you need.

rsync

rsync compares both sides and sends only files that are missing or changed. That matters when you pull outputs several times while a job is running, or when a transfer drops halfway and you restart it. Most clusters have it, and it is either preinstalled or one install away on macOS and WSL.

$ rsync -avhP vanda:~/proj/h2o-opt/ ~/proj/h2o-opt/

Run the same command twice and the second run finishes almost instantly, because nothing changed:

sending incremental file list

sent 233 bytes  received 14 bytes  494.00 bytes/sec
total size is 600.17K  speedup is 2,429.83

The trailing slash

This is the part of rsync that trips everyone up. Whether the source path ends in / changes the result. Here is the same directory copied without and with the slash:

$ rsync -avh proj/h2o-opt a/
sending incremental file list
h2o-opt/
h2o-opt/h2o.gbw
h2o-opt/h2o.inp
h2o-opt/h2o.out
...
$ rsync -avh proj/h2o-opt/ b/
sending incremental file list
h2o.gbw
h2o.inp
h2o.out
...
$ tree a b
a
└── h2o-opt
    ├── h2o.gbw
    ├── h2o.inp
    ...
b
├── h2o.gbw
├── h2o.inp
...

Without the slash, the directory itself is copied (a/h2o-opt/). With it, the directory's contents land directly in the destination. A slash on the destination makes no difference. A habit that avoids surprises: put a slash on both and give the destination the same name as the source. rsync -avh src/ dst/ then reads as "make dst match the contents of src".

Pulling only some files

Often you want just the .out and .xyz files from a set of job directories. Combine --include and --exclude. Rules are checked in order and the first match wins.

$ rsync -avh --dry-run --include='*/' --include='*.out' --include='*.xyz' --exclude='*' \
    vanda:~/proj/ ~/proj/
sending incremental file list
created directory /home/kim/proj
./
h2o-opt/
h2o-opt/h2o.out
h2o-opt/h2o.xyz
h2o-opt/h2o_trj.xyz
nh3-opt/
nh3-opt/nh3.out
nh3-opt/nh3.xyz
nh3-opt/nh3_trj.xyz

sent 264 bytes  received 69 bytes  666.00 bytes/sec
total size is 600.17K  speedup is 1,802.31 (DRY RUN)

--include='*/' lets rsync descend into subdirectories, the next two rules keep matching files, and --exclude='*' drops everything else. Put --exclude='*' first and nothing gets through. Quote the patterns so the shell does not expand them first. When the list looks right, run it again without --dry-run.

To take everything except a few large file types, use --exclude on its own:

$ rsync -avhP --exclude='*.gbw' --exclude='*.tmp' vanda:~/proj/h2o-opt/ ~/proj/h2o-opt/

What not to copy

Besides results, calculations leave large intermediate files: ORCA's .gbw (wavefunction) and .tmp files, the wavefunction files in Quantum ESPRESSO's outdir, big LAMMPS dump files. They can run to several GB and you rarely need them for analysis. What you usually want is the text output (.out, log.lammps) and the structures (.xyz).

Check sizes on the cluster before copying. du -sh gives the size of each directory:

$ du -sh proj/*
7.0M	proj/h2o-opt
7.0M	proj/nh3-opt

A .gbw can serve as the starting guess for the next calculation, so even if you do not download it, leave it on the cluster until you are sure the project is finished.

Many small files: tar

Sending thousands of files one by one is slow because of the per-file overhead. Bundle and compress them with tar on the cluster, transfer the single archive, then unpack it.

# on the cluster: bundle, leaving out large scratch files
$ tar czf h2o-opt.tar.gz --exclude='*.gbw' --exclude='*.tmp' h2o-opt
$ tar tzf h2o-opt.tar.gz
h2o-opt/
h2o-opt/h2o_trj.xyz
h2o-opt/h2o.out
h2o-opt/h2o.inp
h2o-opt/h2o.xyz

# on the laptop: fetch and unpack
$ rsync -avhP vanda:~/proj/h2o-opt.tar.gz .
$ tar xzf h2o-opt.tar.gz

c creates, t lists, x extracts, z means gzip, and f says a file name follows. Listing with tar tzf before extracting tells you whether the archive will scatter files into your current directory. Text output compresses well.

GUI programs

If you prefer drag and drop, use an SFTP client. They all run over ssh, so you log in with the same account and key.

Filtering and resuming are awkward in a GUI. If you pull from many job directories regularly, save the rsync command in a script (shell scripting guide).