Phase 3 System

Isambard Phase 3, just like the MACS, hosts many nodes of different architectures:

Phase 3 nodes run Red Hat Enterprise Linux 8 with the Cray software stack.

All nodes are connected via Slingshot 10. The login nodes are connected to the Internet via a 10 Gigabit link to the Janet Network.

Compiling

Compiling can be perfomed on the p3-login node. Please load the IntelOneApi module for both mpi and compiler

module load IntelOneApi/compiler
module load IntelOneApi/mpi

This will make the standard Intel tools available - being AMD processors we recommend using advice from AMD which suggests using

icc -march core-avx2

To compile with MPI, the following can be used:

mpiicc -march core-avx2

Running a job

The system can use Intel MPI and Compilers. For example to run a job on a single node the milanq:

qsub -q milanq -l select=1:ncpus=128:mpiprocs=128

To run on 2 nodes you could use:

qsub -q milanq -l select=2:ncpus=128:mpiprocs=128 -l place=scatter:excl

This will place each request on different nodes - since we have hyperthreading enabled it would otherwise place them on the same node.

Then use ssh to laucnh the MPI job

mpirun -launcher ssh hostname