slurm, mpi, guix and plafrim

1.1.1. Default usage (without guix)

srun -p hpc /bin/hostname

miriel023.formation.cluster

srun -p hpc -l /bin/hostname

0: miriel023.formation.cluster

srun -l -p hpc -N 2 hostname

scontrol show node miriel017

srun -p hpc -N 2 --ntasks-per-node=24 -l /bin/hostname | sort -n

srun -p hpc -N 2 --ntasks-per-node=24 --pty bash

1.1.2. guix-based usage

srun --version

slurm 17.11.3-2

1. Using srun provided by guix environment
   

   Instead of the one provided by default on plafrim (slurm 17.11.3-2), we can let guix provide it. There is a risk though: the slurm client provided by guix must be able to speak with the slurm daemon (see overview architecture). In our case, this should be fine, guix providing the 17.11.3 version:

   guix environment --pure --ad-hoc slurm -- srun --version
   

   slurm 17.11.3
   

   We can therefore check out the hostname with the srun command provided by guix:

   guix environment --pure --ad-hoc slurm -- srun -l -p hpc -N 2 /bin/hostname
   

   0:	miriel017.formation.cluster
   1:	miriel018.formation.cluster

   Or, even better, check out the hostname with both the srun and hostname (through the inetutils package for the latter) commands provided by guix:

   guix environment --pure --ad-hoc slurm inetutils -- srun -l -p hpc -N 2 hostname
   

   0:	miriel017.formation.cluster
   1:	miriel018.formation.cluster

   In this workflow, as the --pure option cleans up the environment before srun is invoked, slurm environment variables are naturally kept:

   guix environment --pure --ad-hoc slurm coreutils -- srun -l -p hpc -N 2 env | grep SLURM_JOB_NODELIST
   

   0:	SLURMJOBNODELIST=miriel[017-018]
   1:	SLURMJOBNODELIST=miriel[017-018]

2. Using default srun provided by plafrim for launching a guix environment
   

   If slurm was deployed by system administrators independently from guix as it is likely to be the case as of today on a production cluster or supercomputer, and this is the case on plafrim, one may want to rely on the srun command to make sure the client and the dameon are well interfaced. It is still possible to deploy a guix environment in this case as follows. A first option is to provide the full path /usr/bin/srun:

   guix environment --pure --ad-hoc inetutils -- /usr/bin/srun -l -p hpc -N 2 hostname
   

   0:	miriel006.formation.cluster
   1:	miriel009.formation.cluster

   This might be fully satisfactory when using the --pure option and one may decide to rely on it during the hands-on sessions of the school.

   However, if this option does clean up the shell environment in a manner equivalent to env -i (see the env manpage with man env), it does not fully isolate the process. Sometimes it is desirable to isolate the environment as much as possible, for maximal purity and reproducibility. In particular, it may be desirable to prevent access to /usr/bin and other system-wide resources from the development environment. The --container (or -C) option provided by guix environment allows one to do so by relying on linux containers (lxc). You may read this for further information. This feature requires linux-libre 3.19 or newer and can be activated as discussed here (or there) if you deploy yourself guix. Unfortunately, plafrim currently provides an older version (3.10) of the linux kernel. Here is for information the report file of system disk space usage on a machine providing the mechanism:

   guix environment --ad-hoc --container coreutils -- df
   

   Filesystem	1K-blocks	Used	Available	Use%	Mounted	on
   /dev
   /dev/tty
   /dev/shm
   /gnu/store/26sq3rhvxydqpmgfn5kicd11drib0xpd-profile

   One may view that /usr/ and /etc/ cannot interfere in such a configuration.

   Because the --container option is currently not available on plafrim we pursue hour discussion using the --pure option (but the --container one shall be used instead on systems where it is available if one wants to ensure reproducibility at an even higher level).

   srun -l -p hpc -N 2 guix environment --pure --ad-hoc inetutils -- hostname
   

   1:	miriel018.formation.cluster
   0:	miriel017.formation.cluster

   In this workflow, as the --pure (or --container) option cleans up the environment after srun is invoked, slurm environment variables are lost:

   srun -l -p hpc -N 2 guix environment --pure --ad-hoc slurm coreutils -- env | grep SLURM_JOB_NODELIST
   

   If it is not a problem when the goal is simply to display the hostname, it may however be critical in more realistic situations. We can then preserve those variables with the --preserve guix environment option, which takes in argument a regexp, ^SLURM in the present case to preserve slurm environment variables starting with "SLURM":

   srun -l -p hpc -N 2 guix environment --pure --preserve=^SLURM --ad-hoc slurm coreutils -- env | grep SLURM_JOB_NODELIST
   

   0:	SLURMJOBNODELIST=miriel[017-018]
   1:	SLURMJOBNODELIST=miriel[017-018]

   As above, it is possible to enter a bash interactive session, but this time within a guix environment, with bash provided by guix.

   srun -N 2 -p hpc --pty guix environment --pure --ad-hoc slurm bash -- bash --norc
   

   Yet, we'll discuss below how to use salloc with guix, which may be more convenient to conduct such interactive sessions.

   To illustrate once again the symmetry of both behaviours, let us dig further in the hardware topology using hwloc, using either srun within a guix environment:

   guix environment --pure --ad-hoc slurm hwloc -- srun -p hpc hwloc-ls
   

   (result omitted) or, equivalently, guix environment within srun:

   mistral01.formation.cluster
   

   srun -p hpc guix environment --pure --ad-hoc hwloc -- hwloc-ls
   

   Machine (128GB total)
   0
   0 (15MB)
   0 (P#0 32GB)
   0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#0)
   1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1 (P#2)
   2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2 (P#4)
   3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3 (P#6)
   4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4 (P#8)
   5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5 (P#10)
         HostBridge
           PCIBridge
             PCI 03:00.0 (InfiniBand)
               Net "ib0"
               OpenFabrics "qib0"
           PCIBridge
             PCI 01:00.0 (Ethernet)
               Net "em1"
             PCI 01:00.1 (Ethernet)
               Net "em2"
           PCI 00:11.4 (SATA)
             Block(Disk) "sda"
           PCIBridge
             PCI 06:00.0 (Ethernet)
               Net "em3"
             PCI 06:00.1 (Ethernet)
               Net "em4"
           PCIBridge
             PCIBridge
               PCIBridge
                 PCIBridge
                   PCI 0a:00.0 (VGA)
           PCI 00:1f.2 (SATA)
   1 (15MB)
   1 (P#2 32GB)
   6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6 (P#12)
   7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7 (P#14)
   8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8 (P#16)
   9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9 (P#18)
   10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10 + PU L#10 (P#20)
   11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11 + PU L#11 (P#22)
   1
   2 (15MB)
   2 (P#1 32GB)
   12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12 + PU L#12 (P#1)
   13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13 + PU L#13 (P#3)
   14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14 + PU L#14 (P#5)
   15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15 + PU L#15 (P#7)
   16 (256KB) + L1d L#16 (32KB) + L1i L#16 (32KB) + Core L#16 + PU L#16 (P#9)
   17 (256KB) + L1d L#17 (32KB) + L1i L#17 (32KB) + Core L#17 + PU L#17 (P#11)
         HostBridge
           PCIBridge
             PCI 82:00.0 (Fabric)
               Net "ib1"
               OpenFabrics "hfi1_0"
   3 (15MB)
   3 (P#3 32GB)
   18 (256KB) + L1d L#18 (32KB) + L1i L#18 (32KB) + Core L#18 + PU L#18 (P#13)
   19 (256KB) + L1d L#19 (32KB) + L1i L#19 (32KB) + Core L#19 + PU L#19 (P#15)
   20 (256KB) + L1d L#20 (32KB) + L1i L#20 (32KB) + Core L#20 + PU L#20 (P#17)
   21 (256KB) + L1d L#21 (32KB) + L1i L#21 (32KB) + Core L#21 + PU L#21 (P#19)
   22 (256KB) + L1d L#22 (32KB) + L1i L#22 (32KB) + Core L#22 + PU L#22 (P#21)
   23 (256KB) + L1d L#23 (32KB) + L1i L#23 (32KB) + Core L#23 + PU L#23 (P#23)
   

1.2.1. Default usage (without guix)

salloc -p hpc -N 2 -n 4
srun -l /bin/hostname | sort -n

hostname

mistral01.formation.cluster

srun -l /bin/hostname | sort -n

squeue -u $USER

exit

salloc: Relinquishing job allocation 426762

scancel 426762

salloc: Job allocation 426762 has been revoked.

1.2.2. guix-based usage

1. Using srun provided by guix environment
   
2. Using default srun provided by plafrim for launching a guix environment
   

1.3.1. Two lines

#!/bin/sh
#SBATCH --time=03:59:00
#SBATCH -N 2
#SBATCH -n 4
#SBATCH -p hpc
guix environment --pure --ad-hoc hello -- /bin/bash --norc
srun -l --mpi=pmi2 hello

srun: error: miriel003: task 3: Exited with exit code 2
0: slurmstepd: error: execve(): hello: No such file or directory
3: slurmstepd: error: execve(): hello: No such file or directory
1: slurmstepd: error: execve(): hello: No such file or directory
2: slurmstepd: error: execve(): hello: No such file or directory
srun: error: miriel002: tasks 0-2: Exited with exit code 2

1.3.2. One line

#!/bin/sh
#SBATCH --time=03:59:00
#SBATCH -N 2
#SBATCH -n 4
#SBATCH -p hpc
srun -l --mpi=pmi2  guix environment --pure --ad-hoc hello -- hello

0: Hello, world!
2: Hello, world!
1: Hello, world!
3: Hello, world!

1.3.3. Two files (to be completed)

#!/bin/sh
#SBATCH --time=03:59:00
#SBATCH -N 2
#SBATCH -n 4
#SBATCH -p hpc
guix environment --pure --ad-hoc hello -- hello.sh

#!/bin/bash
mpirun --mpi=pmi2 hello

4.2.1. Many ways to use the network

4.2.2. How drivers are selected

"configure: error: PSM support requested but not found. Aborting"

4.2.3. Verifying what was enabled/compiled

Transports
-----------------------
Cisco usNIC: no
Cray uGNI (Gemini/Aries): no
Intel Omnipath (PSM2): yes     <- miriel001-043
Intel TrueScale (PSM): yes     <- miriel001-088
Mellanox MXM: no
Open UCX: yes                  <- mistral+sirocco01-06
OpenFabrics OFI Libfabric: no  <- explicitly disabled
OpenFabrics Verbs: no          <- explicitly disabled
Portals4: no
Shared memory/copy in+copy out: yes
Shared memory/Linux CMA: yes
Shared memory/Linux KNEM: no
Shared memory/XPMEM: no
TCP: yes

$ ompi_info | grep ucx
MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v4.0.2)
MCA pml: ucx (MCA v2.1.0, API v2.0.0, Component v4.0.2)
$ ompi_info | grep psm
MCA mtl: psm (MCA v2.1.0, API v2.0.0, Component v4.0.2)
MCA mtl: psm2 (MCA v2.1.0, API v2.0.0, Component v4.0.2)
$ ompi_info | grep Configure
Configure command line: '--with-ucx' '--with-psm' '--with-psm2' '--disable-verbs' '--without-ofi' '--enable-mpirun-prefix-by-default' '--prefix=...'

4.2.4. Performance benchmarks to check whether the right network is used

make IMB-MPI1 CC=/my/ompi/bin/mpicc CXX=/my/ompi/bin/mpicxx

mpiexec -np 2 -H mistral02,mistral03 --map-by node --bind-to core IMB-MPI1 Pingpong

1. PSM2
   

   On miriel001-043, we want to get 100Gbit/s bandwidth. That's supposed to be 12.5GB/s but we rather get 10GB/s in practice because some bandwidth is lost in packet headers, signalling bits, etc.

   $ mpiexec -np 2 --map-by node --bind-to core ... IMB-MPI1 Pingpong
            size         iter      latency    bandwidth
               0         1000         1.45         0.00
               1         1000         1.62         0.62
   [...]
         2097152           20       215.75      9720.19
         4194304           10       439.37      9546.20
   

   If we add --mca mtl ^psm2 to disable PSM2, or --mca mtl psm to force PSM1 only, the performance disappears (see below). Hence we were really using PSM2 here, and getting (almost) the expected 10GB/s.

2. PSM1
   

   On miriel001-088, we want to get 40Gbit/s (when we cannot use the 100G PSM2 network).

   $ mpiexec -np 2 --map-by node --bind-to core --mca mtl psm ... IMB-MPI1 Pingpong
            size         iter      latency    bandwidth
               0         1000         1.56         0.00
               1         1000         1.68         0.59
   [...]
         2097152           20       689.57      3041.23
         4194304           10      1325.47      3164.39
   

   If we add --mca mtl ^psm2,psm to disable PSM2 and PSM1, the performance disappears. Hence we were really using PSM1 here, and getting (almost) the expected 4GB/s.

3. TCP
   

   Still on miriel, if we force use TCP on the 10G Ethernet network by disabling PSM1 and PSM2, we get about 1GB/s as expected:

   $ mpiexec -np 2 --map-by node --bind-to core --mca mtl ^psm2,psm ... IMB-MPI1 Pingpong
            size         iter      latency    bandwidth
               0         1000        16.39         0.00
               1         1000        18.08         0.06
   [...]
         2097152           20      2314.69       906.02
         4194304           10      5323.38       787.90
   

   In this case, OpenMPI is using the TCP BTL. The equivalent configuration is --mca btl tcp,vader,self. The vader BTL is here for communication between processes on the same node (there are some StarWars-related jokes in OpenMPI component names). The self is for communication between a process and itself.

4. UCX
   

   Now we switch to mistral nodes with 40G Mellanox InfiniBand. The default configuration gives almost 4GB/s as expected.

   $ mpiexec -np 2 --map-by node --bind-to core ... IMB-MPI1 Pingpong
            size         iter      latency    bandwidth
               0         1000         1.64         0.00
               1         1000         1.66         0.60
         2097152           20       567.42      3695.97
         4194304           10      1126.84      3722.18
   

   If we disable UCX with --mca pml ^ucx, performance disappears as expected. It switches to TCP.

5. OpenIB (deprecated Verbs driver)
   

   If we didn't disable Verbs support with --disable-verbs, it doesn't get used unless forced. The way to force it is to disable UCX and pass --mca btl_openib_allow_ib 1 (to allow InfiniBand cards to be used in the openib BTL). This is equivalent to passing --mca btl openib,vader,self to OpenMPI before version 4.

   $ mpiexec -np 2 --map-by node --bind-to core --mca pml ^ucx --mca btl_openib_allow_ib 1 ... IMB-MPI1 Pingpong
            size         iter      latency    bandwidth
               0         1000         1.48         0.00
               1         1000         1.54         0.65
   [...]
         2097152           20       576.53      3637.52
         4194304           10      1128.36      3717.16
   

   The performance is actually very good, basically identical to UCX, so it is not obvious why Mellanox wants to deprecated openib support in OpenMPI. However this hardware is old. Maybe the difference would be larger on recent Mellanox hardware.

5.2.1. Long jobs