Set up
Table of Contents
1. Working in pairs
01. Shilova/Thibault - 02. Cirrottola 03. Citrain - 04. Schenkels 05. Jacquet - 06. Mirza 07. Labat - 08. Peressoni 09. Meyer - 10. Lion 11. Rouxelin - 12. Ravel 13. Sirdey - 14. Carlino 15. Vasanthan - 16. Joshi 17. Bellezza - 18. Lecouvez 19. Fondanèche - 20. Messai 21. Froehly - 22. Korkmaz 23. Furmento - 24. Xiang 25. Felsoci - 26. Kazolea 27. Lux - 28. Ciallella 29. Poette - 30. Iannacito
We invite you to get set up in pairs with team mates from a different background. Here is a suggestion. Both members of the team can do all the coding and experimental work redundantly, but this will give you the opportunity to exchange in case you get stuck on a point you are not familiar with.
Note the associated individual number, you will use it later on in the session.
2. Connection to plafrim formation
Most of the work we'll do this week will take place on plafrim.
We'll rely on ssh to connect to the platform. You have received
on Thursday October 31 an email from plafrim support with
instructions to connect to the machine, together with a password.
They must look like where <myname> is to be changed according
to your login instructions:
ssh hpcs-<myname>@formation.plafrim.fr ssh plafrim
After these two steps, you should be on the mistral01 node. You can check that
with the following instruction:
hostname
mistral01.formation.cluster
These two steps can be gathered into a single step. To do so, You can add the
following lines in your .ssh/config, where <myname> is to be changed
according to your login instructions:
Host plafrim-hpcs
ForwardAgent yes
ForwardX11 yes
User hpcs-<myname>
ProxyCommand ssh -T -q -o "ForwardAgent yes" -l hpcs-<myname> formation.plafrim.fr 'ssh-add -t 10 && nc plafrim 22'
You can then log onto plafrim-hpcs reaching
ssh plafrim-hpcs
Note that you can use the -X, or, even better -Y, ssh option to enable
regular or trusted X11 forwarding, respectively:
ssh -Y plafrim-hpcs
2.1. Troubleshooting
2.1.1. -Y issue
The -Y trusted option may require further authentication setup. You may decide
not to use it at all (it's not requested in our hands-on) or use -X.
2.1.2. Authentication agent issue
On some systems where ssh-agent is not set up by default you may get the following error:
Could not open a connection to your authentication agent. kex_exchange_identification: Connection closed by remote host
If ssh-agent is not automatically started at login, it can be started manually with the command (see details):
eval `ssh-agent`
2.1.3. Lifetime issue
The ssh-agent default lifetime may not be sufficient resulting in a message
of the form:
Lifetime set to 1 seconds hpcs-<myname>@plafrim-hpcs's password: Permission denied, please try again.
In this case, you may edit your .ssh/config file and change the ssh-add -t
10 in the ProxyCommand to ssh-add -t 10 before saving and trying to
reconnect.
2.1.4. ssh key (turned off on plafrim-hpcs)
One may want to use an ssh key to avoid typing the password at
each login. In principle, following the ANSSI recommendations, an
rsa key of at least 2048 bits should be used. The ssh pair
of keys could be be generated as follows where <rsa-filename>
is /.ssh/id_rsa by default:
ssh-keygen -t rsa -b 2048 -f <rsa-filename>
However, ssh identification through ssh key is turned off on
plafrim-hpcs and you must keep relying on password authentication.
3. Set up of the environment with guix
One of the objective of this school is to allow you mastering your environment. We will use guix to handle that. Guix is an advanced distribution of the GNU operating system developed by the GNU Project, which respects the freedom of computer users. It is a transactional package manager, with support for per-user package installations. Users can install their own packages without interfering with each other, yet without unnecessarily increasing disk usage or rebuilding every package.
Thanks to joint effort of the guix development and plafrim team, guix is
readily available on plafrim as detailed here.
The guix-hpc initiative is a solid foundation for hpc and reproducible science.
The software environments created with guix are fully reproducible: a package
built from a specific guix commit on your laptop will be exactly the same as
the one built on the HPC cluster – plafrim in our case – you deploy it to,
usually bit-for-bit.
Guix and its package collection are updated by running guix pull (see Invoking
guix pull). By default guix pull downloads and deploys guix itself from the
official GNU Guix repository. This can be customized by defining channels in the
~/.config/guix/channels.scm file. A channel specifies a URL and branch of a
Git repository to be deployed, and guix pull can be instructed to pull from
one or more channels. In other words, channels can be used to customize and to
extend guix.
Here is the proposed content we suggest (or, more accurately, strongly encourage) you to use for this hands-on session:
(list (channel
(name 'guix-hpc-non-free)
(url "https://gitlab.inria.fr/guix-hpc/guix-hpc-non-free.git")
(commit
"0822b7eebb63e2360125e017934ba817686d9669"))
(channel
(name 'guix-hpc)
(url "https://gitlab.inria.fr/guix-hpc/guix-hpc.git")
(commit
"dd37a4cf3c4e48b8e1b404e355c48f5ae1a6d424"))
(channel
(name 'guix)
(url "https://git.savannah.gnu.org/git/guix.git")
(commit
"4c463569b7af3ee064d323691bc0284155f9a85d")))
You can fill your /.config/guix/channels.scm accordingly by editing it,
copying the above text and paste. Alternatively, as on plafrim-hpcs, this file
is already available at /home/hpcs-readonly/channels.scm, you can directly
copy it as follows (you may need to make the /.config/guix/ directory before):
cp /home/hpcs-readonly/channels.scm ~/.config/guix/channels.scm
If it is the first time you are invoking guix on the platform, once you have
set up channels.scm, you must first do the following initialization:
guix build hello
After that, you can invoke guix pull:
guix pull
You are all set up for performing HPC in a reproducible set up. Well … let's see!
4. Going further
4.1. python environment set up
Set up your python environment.
4.2. slurm and mpi
Set up your slurm and mpi environment.