Background

Parallelized software that runs on a compute cluster must execute multiple copies of its program across the allocated compute nodes in order to achieve parallel (simultaneous) computation. Most parallel software packages on NOTS rely on SLURM's srun command to perform this task, but some packages, such as Gaussian Linda, certain MPI libraries (via their own mpirun command), and others, will want to directly execute programs on the compute nodes via SSH. This behavior is allowed, but requires some extra setup on the user's part.

You might need to perform these steps if your program does not use srun and experiences network errors, hangs, or time outs near the beginning of execution, during the time when it attempts to spawn its program to the other nodes.

"Passwordless" SSH

The setup steps below enable "passwordless" SSH, which allows you (or a program running on your behalf) to use SSH to connect from one node of the cluster to another node, without requiring you to enter your NetID password. (Doing so does not necessarily affect how you log into the cluster's login nodes from outside of the cluster.) The passwordless feature is necessary to allow parallel jobs you submit to run unattended.

Create Host Key

The first step in establishing passwordless SSH is to create your public host key. Login to the cluster and run the ssh-keygen command as shown below. Accept all of the default values and do not enter a passphrase.

ssh-keygen example

Add Host Key to authorized_keys File

After you have created your public host key above, append the contents of ~/.ssh/id_rsa.pub to ~/.ssh/authorized_keys. This step will enable programs such as mpirun to login from one compute node to another using SSH without a password.

If you do not have an "authorized_keys" file in your ~/.ssh directory, you will need to create one like so:

[beaker@loginx1 ~]$ touch ~/.ssh/authorized_keys

[beaker@loginx1 ~]$ chmod 600 ~/.ssh/authorized_keys

Once it exists, copy the public host key into it like so:

[beaker@loginx1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Disable Strict Host Key Checking

To avoid SSH prompts when automatically logging into compute nodes allocated by the scheduler, configure SSH to not use strict host key checking. Create the file ~/.ssh/config with contents as shown below.

~/.ssh/config file example

This configuration will also suppress creation of entries in ~/.ssh/known_hosts and reduce log messages.