ssh and keys

What ssh is

ssh (secure shell) gives you a shell on another computer over the network. Run ssh on your laptop and a shell opens on the cluster's login node; every command you type from then on runs on the cluster. The connection is encrypted.

Your ORCA and Quantum ESPRESSO jobs run on the cluster, not on your laptop. Uploading inputs, submitting jobs and checking outputs all happen over an ssh connection, and the scp and rsync commands in the next chapter use ssh underneath. Linux, macOS and Windows 10 onwards all ship an ssh client.

First login

The form is ssh username@host. The host is the address of the cluster's login node, which you get along with your account. The address and username below are examples.

$ ssh e0123456@vanda.nus.edu.sg

The first time you connect to a server, ssh asks this:

The authenticity of host 'vanda.nus.edu.sg (...)' can't be established.
ED25519 key fingerprint is SHA256:...
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'vanda.nus.edu.sg' (ED25519) to the list of known hosts.
e0123456@vanda.nus.edu.sg's password:

Every server has its own host key, and the fingerprint is a short summary of it. Answering yes records it in ~/.ssh/known_hosts, and ssh will not ask again. If the cluster documentation publishes its fingerprints, compare before you answer. Nothing appears on screen while you type the password; that is normal.

Once you are in, the prompt changes to the cluster's. Leave with exit or Ctrl-D. From off campus you usually need to connect to the university VPN first; follow your cluster's instructions.

Creating a key

Instead of a password you can authenticate with a key pair. A key pair is two files. The private key (id_ed25519) stays on your computer; the public key (id_ed25519.pub) goes on the server, which uses it to check that you hold the private key. You create the pair once, on your laptop (or in WSL).

$ ssh-keygen -t ed25519 -C "kim@laptop"
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/kim/.ssh/id_ed25519):
Enter passphrase for "/home/kim/.ssh/id_ed25519" (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/kim/.ssh/id_ed25519
Your public key has been saved in /home/kim/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:zQ5PSX1x2MqziJIYvYB5ZNZXFsTEXyfTCAD3WJLq+nM kim@laptop

Press Enter to accept the default location. The text after -C is a label stored in the key, so you can tell later which machine it came from. The passphrase encrypts the private key file: if your laptop is lost, the key is useless without it. Set one. ssh-agent, below, saves you from typing it every time.

$ cat ~/.ssh/id_ed25519.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJE3ZbxNBjHILtE8MNFKpTzBNAjrxmjLkruM0m/mfZLw kim@laptop

Putting the public key on the server

The server accepts any public key listed in its ~/.ssh/authorized_keys. On Linux, macOS and WSL, ssh-copy-id adds it for you. It asks for your password once and reports Number of key(s) added: 1 when done.

$ ssh-copy-id -i ~/.ssh/id_ed25519.pub e0123456@vanda.nus.edu.sg

Windows PowerShell has no ssh-copy-id. Pipe the public key over ssh and append it on the server side instead:

# one line, in PowerShell
type $env:USERPROFILE\.ssh\id_ed25519.pub | ssh e0123456@vanda.nus.edu.sg "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

Now ssh e0123456@vanda.nus.edu.sg asks for your key's passphrase (if you set one) instead of the password. Some clusters disable key login or require a second factor on top; if it does not work, check the documentation first.

ssh refuses key files with loose permissions. ~/.ssh must be 700, and the private key, config and authorized_keys 600 (see chapter 04). This applies on both the laptop and the server.

$ chmod 700 ~/.ssh
$ chmod 600 ~/.ssh/id_ed25519 ~/.ssh/config
$ ls -la ~/.ssh
total 12
drwx------ 2 kim kim 100 Sep 29 00:12 .
drwxrwxr-x 5 kim kim 120 Sep 29 00:09 ..
-rw------- 1 kim kim 202 Sep 29 00:12 config
-rw------- 1 kim kim 444 Sep 29 00:07 id_ed25519
-rw-r--r-- 1 kim kim  92 Sep 29 00:07 id_ed25519.pub

~/.ssh/config

Typing e0123456@vanda.nus.edu.sg twenty times a day gets old. Put an alias in ~/.ssh/config on your laptop and ssh vanda is enough. Create the file if it does not exist.

Host vanda
    HostName vanda.nus.edu.sg
    User e0123456
    IdentityFile ~/.ssh/id_ed25519
    ServerAliveInterval 60

The alias works in everything that uses ssh: scp h2o.inp vanda:~/proj/, rsync ... vanda:..., and VS Code Remote-SSH all read this file. ssh -G shows what ssh actually resolved:

$ ssh -G vanda | grep -E '^(hostname|user|identityfile) '
user e0123456
hostname vanda.nus.edu.sg
identityfile ~/.ssh/id_ed25519

Some machines are not reachable from outside and have to be reached through a gateway. Put the gateway in ProxyJump and ssh chains the two hops for you. On the command line the same thing is ssh -J vanda e0123456@inner.example.org.

Host inner
    HostName inner.example.org
    User e0123456
    ProxyJump vanda

ssh-agent

ssh-agent holds your unlocked private key in memory and hands it to ssh when needed, so you type the passphrase once per session. macOS and desktop Linux usually have one running already. In WSL you start it yourself:

$ eval "$(ssh-agent -s)"
Agent pid 454122
$ ssh-add ~/.ssh/id_ed25519
Enter passphrase for /home/kim/.ssh/id_ed25519:
Identity added: /home/kim/.ssh/id_ed25519 (kim@laptop)
$ ssh-add -l
256 SHA256:zQ5PSX1x2MqziJIYvYB5ZNZXFsTEXyfTCAD3WJLq+nM kim@laptop (ED25519)

A Host * block at the bottom of ~/.ssh/config containing AddKeysToAgent yes adds the key to the agent automatically the first time you unlock it.

Common errors

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
...
Offending ED25519 key in /home/kim/.ssh/known_hosts:3

This appears when a cluster is reinstalled or a login node replaced, because the host key changes. Rarely, it really is someone intercepting the connection. Confirm with the cluster announcements or the admins that the key changed, then delete the old entry and reconnect to accept the new fingerprint.

$ ssh-keygen -R vanda.nus.edu.sg
# Host vanda.nus.edu.sg found: line 3
/home/kim/.ssh/known_hosts updated.
Original contents retained as /home/kim/.ssh/known_hosts.old

X11 forwarding and VS Code

To see windows from programs running on the cluster (VMD, a gnuplot window) on your laptop, connect with ssh -Y vanda. macOS needs XQuartz; WSL on Windows 11 often works without extra setup. The screen is drawn over the network, so it is slow. For a large trajectory, copy the file back and open it locally.

VS Code's Remote-SSH extension connects using the aliases in ~/.ssh/config and lets you edit cluster files as if they were local, which is handy for input files. It does start a server process on the login node, and some clusters restrict it for that reason.