Python environment
Table of Contents
We assume that you are all set up with your plafrim
access and guix
environment.
Once you have set up your guix
environment on plafrim, we propose two options
to deploy a python environment. You can choose any of them or, if you want, play
with both.
If you want to go straight to the point and skip for now the below discussion on
possible frameworks to work on a python environment on plafrim
with guix
,
just do
ssh -L 500<XY>:localhost:500<XY> plafrim-hpcs guix environment --pure --ad-hoc jupyter python-ipython python-ipykernel python-matplotlib python-pandas python-scipy python-slurm-magic bash wget coreutils -- jupyter notebook --ip=127.0.0.1 --port=500<XY> --no-browser --NotebookApp.notebook_dir=/home/
where <XY> is your personal id (two digits) and you are all set up to
load the hdg-step0.ipynb
notebook provided on the server under the
repository /home/hpcs-readonly/notebook
and play with the instructions
given within it.
If you want to further dig into the possibilities for setting a python environment, feel free to read the following.
1. Interpreter
We assume you have log into the platform:
ssh plafrim-hpcs
You shall now be on the frontal node (mistral01
)
hostname
mistral01.formation.cluster
You can play with an editor (vim
in the following, but you can use another
one such as emacs
if you prefer) and the python
interpreter. Here is how to
obtain such a guix environment:
guix environment --pure --ad-hoc python python-ipython python-scipy coreutils vim -- /bin/bash --norc
Note that we need python-scipy.
For instance, after fetching the hdg.py file, you may make the following test
(we assume you have made a python
repository to work in, with e.g. mkdir
python
):
cd python
python3 hdg.py
Note that this file is to be completed hence, beyond the following expected warning, you are not expected to have a working python file.
For those who prefer a matlab/octave
environment, you can instead try:
guix environment --pure --ad-hoc octave coreutils vim -- /bin/bash --norc
For instance, fetching the hdg.m file, you may make the following test (we
assume you have made a matlab
repository to work in, with e.g. mkdir
matlab
):
cd matlab
octave hdg.m
2. Jupyter
Alternatively, you can deploy a jupyter notebook.
2.1. A first idea (not fully convenient)
guix environment --pure --ad-hoc jupyter python-ipython python-ipykernel python-matplotlib python-pandas python-scipy python-slurm-magic bash wget coreutils -- jupyter notebook --NotebookApp.notebook_dir=/home/
Note that, when in this notebook context, the python
source code must be
preceded (line 1) by %matplotlib notebook
in order to get a proper display.
2.2. A better framework: your laptop browser with a plafrim backend
The drawback of the above approach is that we need to launch a browser on plafrim. We want to use a browser on our own laptop while plafrim is taking in charge the computational load. We propose the following set up to do so.
Assuming you are on your laptop session (hence possibly after logging out from
plafrim
). We create an ssh
tunnel to redirect the port 8000 of your laptop
to the port 500<XY> of plafrim where you must set up <XY>
as your personal
id (two digits between 01 and 30).
ssh -L 500<XY>:localhost:500<XY> plafrim-hpcs
From this ssh
tunnel we can now load the guix
environment, with the selected
above port, with the following command where you have to set up <XY>
once
again as your personal id, two digits between 01 and 30:
guix environment --pure --ad-hoc jupyter python-ipython python-ipykernel python-matplotlib python-pandas python-scipy python-slurm-magic bash wget coreutils -- jupyter notebook --ip=127.0.0.1 --port=500<XY> --no-browser --NotebookApp.notebook_dir=/home/
You will obtain a token of the form:
[...] Or copy and paste one of these URLs: http://127.0.0.1:500<XY>/?token=<mytoken>
You can copy the obtained http://127.0.0.1:500<XY>/?token=<mytoken>
url
and
paste it to your browser's bar address of your laptop, or, execute directly the
following instruction, assuming your browser is firefox
, with the actual url
you obained:
firefox http://127.0.0.1:500<XY>/?token=<mytoken>
2.3. Play with an actual notebook
You may want to play with an actual notebook. You can either make a new one or get an existing one that you upload. Do not hesitate to have a quick look at the jupyter reference. In the following, we will propose you to upload and work with this notebook, which remains to be completed!
Note that the --NotebookApp.notebook_dir=/home/
option removes the restriction
imposed by the jupyter server of browsing within our home directory, as
discussed here. You can thus have directly access to
/home/hpcs-readonly/notebook
and load the hdg-step0.ipynb
notebook (without
having to download it and upload it). As it names pretends, this is a read only
repository. Therefore you may not save your file in-place but use the save as
option instead. You will be prompted to "Enter a notebook path relative to
notebook dir". You can enter hpcs-<myname>/
or,
hpcs-<myname>/notebook/mynotebook
(assuming if you already created a
hpcs-<myname>/notebook/
directory), where <myname>
is to be changed.
2.4. Kill the server
The CTRL-C
keystroke is not sufficient to kill the jupyter
server. You
need to further perform:
pkill -9 jupyter
2.5. Notes
2.5.1. Running the notebook on a miriel
compute node
In the above set up, both the notebook server and actual python computation
are running on the mistral01
frontal node. As the amount of computation
is relatively low and that 24 cores are available, this shall not be a
problem in the proposed hands-on session.
However, there are multiple options to do so. We discuss two of them here. Feel
free to try it. Basic knowledge of slurm
is assumed (or read that).
We assume that you are logged into plafrim
as follows (with <XY> properly set
up):
ssh -L 500<XY>:localhost:500<XY> plafrim-hpcs
- Submitting tasks to compute nodes with
slurm-magic
(idea only)
The first option is to rely on slurm-magic to delegate part of the computation to compute nodes through
slurm
. This is not very useful for a purepython
hands-on as the one onhdg 1d
you have been working on Tuesday. But it might be of interest for you in the future to fully exploitplafrim
.In this situation, you launch
jupyter
from the frontal nodemistral01
exactly as above:guix environment --pure --ad-hoc jupyter python-ipython python-ipykernel python-matplotlib python-pandas python-scipy python-slurm-magic bash wget coreutils -- jupyter notebook --ip=127.0.0.1 --port=500<XY> --no-browser --NotebookApp.notebook_dir=/home/
The server is thus running on the frontal node. The idea is to delegate (only) some of the computation (the costly ones) to the
miriel
comptute nodes throughslurm
. We use slurm-magic to do so. Note that in the aboveguix environment
command line, thepython-slurm-magic
guix
package is loaded to provide the service on thejupyter
server side. So there is nothing more to do on the server side in our case.We are almost set to play on the client side (as above within your browser's laptop). Make a new cell (for instance at the top of your notebook, but not necessarily) and load the
slurm-magic
extension:%load_ext slurm_magic
You can do view the available magic commands with (again type it in a jupyter cell and run it):
%lsmagic
Want to view the compute partitions available on plafrim?
%sinfo
We observe that the
hpc
partition onto which we view thatmiriel
compute nodes are available. Let's run on that partition!%srun -p hpc /bin/hostname
If you want to know more (
%%sbatch
, …), we invite you to a look at this more excellent tutorial. - Running both the notebook server and the computation on a compute node (one more
ssh
bounce)
hostname
mistral01.formation.cluster
The second option consists of reserving a compute node and performing an additional
ssh
bounce onto it.We reserve a compute node:
salloc -p hpc
salloc: Granted job allocation 426856
We then check out the node we got allocated:
squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 426856 hpc bash hpcs-agu R INVALID 1 miriel032 We then perform an additional bounce maintaining port redirection:
ssh -L 500<XY>:localhost:500<XY> miriel0<ZZ>
where <XY> and <ZZ> are to be once again changed (you personal digits and the miriel number – 32 – in our case, respectively).
All is set up to start the notebook from the compute node
guix environment --pure --ad-hoc jupyter python-ipython python-ipykernel python-matplotlib python-pandas python-scipy python-slurm-magic bash wget coreutils -- jupyter notebook --ip=127.0.0.1 --port=50034 --no-browser --NotebookApp.notebook_dir=/home/
You may copy and paste the proposed
url
into your laptop's browser. You may want to check out on which node you are running by typing and executing the following command in a cell of your notebook (the!
makes it abash
request instead of apython command
):%srun -p hpc /bin/hostname
The command has been run on a
miriel
node (miriel032
in our case).
2.5.2. Embedding reproducibility in the notebook
You may want to read this guix feature.