Python environment

Table of Contents

We assume that you are all set up with your plafrim access and guix environment.

Once you have set up your guix environment on plafrim, we propose two options to deploy a python environment. You can choose any of them or, if you want, play with both.

If you want to go straight to the point and skip for now the below discussion on possible frameworks to work on a python environment on plafrim with guix, just do

ssh -L 500<XY>:localhost:500<XY> plafrim-hpcs
guix environment --pure --ad-hoc jupyter python-ipython python-ipykernel python-matplotlib python-pandas python-scipy python-slurm-magic bash wget coreutils -- jupyter notebook --ip=127.0.0.1 --port=500<XY> --no-browser --NotebookApp.notebook_dir=/home/

where <XY> is your personal id (two digits) and you are all set up to load the hdg-step0.ipynb notebook provided on the server under the repository /home/hpcs-readonly/notebook and play with the instructions given within it.

If you want to further dig into the possibilities for setting a python environment, feel free to read the following.

1. Interpreter

We assume you have log into the platform:

ssh plafrim-hpcs

You shall now be on the frontal node (mistral01)

hostname
mistral01.formation.cluster

You can play with an editor (vim in the following, but you can use another one such as emacs if you prefer) and the python interpreter. Here is how to obtain such a guix environment:

guix environment --pure --ad-hoc python python-ipython python-scipy coreutils vim -- /bin/bash --norc

Note that we need python-scipy.

For instance, after fetching the hdg.py file, you may make the following test (we assume you have made a python repository to work in, with e.g. mkdir python):

cd python
python3 hdg.py

Note that this file is to be completed hence, beyond the following expected warning, you are not expected to have a working python file.

For those who prefer a matlab/octave environment, you can instead try:

guix environment --pure --ad-hoc octave coreutils vim -- /bin/bash --norc

For instance, fetching the hdg.m file, you may make the following test (we assume you have made a matlab repository to work in, with e.g. mkdir matlab):

cd matlab
octave hdg.m

2. Jupyter

Alternatively, you can deploy a jupyter notebook.

2.1. A first idea (not fully convenient)

guix environment --pure --ad-hoc jupyter python-ipython python-ipykernel python-matplotlib python-pandas python-scipy python-slurm-magic bash wget coreutils -- jupyter notebook --NotebookApp.notebook_dir=/home/

Note that, when in this notebook context, the python source code must be preceded (line 1) by %matplotlib notebook in order to get a proper display.

2.2. A better framework: your laptop browser with a plafrim backend

The drawback of the above approach is that we need to launch a browser on plafrim. We want to use a browser on our own laptop while plafrim is taking in charge the computational load. We propose the following set up to do so.

Assuming you are on your laptop session (hence possibly after logging out from plafrim). We create an ssh tunnel to redirect the port 8000 of your laptop to the port 500<XY> of plafrim where you must set up <XY> as your personal id (two digits between 01 and 30).

ssh -L 500<XY>:localhost:500<XY> plafrim-hpcs

From this ssh tunnel we can now load the guix environment, with the selected above port, with the following command where you have to set up <XY> once again as your personal id, two digits between 01 and 30:

guix environment --pure --ad-hoc jupyter python-ipython python-ipykernel python-matplotlib python-pandas python-scipy python-slurm-magic bash wget coreutils -- jupyter notebook --ip=127.0.0.1 --port=500<XY> --no-browser --NotebookApp.notebook_dir=/home/

You will obtain a token of the form:

[...]
Or copy and paste one of these URLs:
    http://127.0.0.1:500<XY>/?token=<mytoken>

You can copy the obtained http://127.0.0.1:500<XY>/?token=<mytoken> url and paste it to your browser's bar address of your laptop, or, execute directly the following instruction, assuming your browser is firefox, with the actual url you obained:

firefox http://127.0.0.1:500<XY>/?token=<mytoken>

2.3. Play with an actual notebook

You may want to play with an actual notebook. You can either make a new one or get an existing one that you upload. Do not hesitate to have a quick look at the jupyter reference. In the following, we will propose you to upload and work with this notebook, which remains to be completed!

Note that the --NotebookApp.notebook_dir=/home/ option removes the restriction imposed by the jupyter server of browsing within our home directory, as discussed here. You can thus have directly access to /home/hpcs-readonly/notebook and load the hdg-step0.ipynb notebook (without having to download it and upload it). As it names pretends, this is a read only repository. Therefore you may not save your file in-place but use the save as option instead. You will be prompted to "Enter a notebook path relative to notebook dir". You can enter hpcs-<myname>/ or, hpcs-<myname>/notebook/mynotebook (assuming if you already created a hpcs-<myname>/notebook/ directory), where <myname> is to be changed.

2.4. Kill the server

The CTRL-C keystroke is not sufficient to kill the jupyter server. You need to further perform:

pkill -9 jupyter

2.5. Notes

2.5.1. Running the notebook on a miriel compute node

In the above set up, both the notebook server and actual python computation are running on the mistral01 frontal node. As the amount of computation is relatively low and that 24 cores are available, this shall not be a problem in the proposed hands-on session.

However, there are multiple options to do so. We discuss two of them here. Feel free to try it. Basic knowledge of slurm is assumed (or read that).

We assume that you are logged into plafrim as follows (with <XY> properly set up):

ssh -L 500<XY>:localhost:500<XY> plafrim-hpcs
  1. Submitting tasks to compute nodes with slurm-magic (idea only)

    The first option is to rely on slurm-magic to delegate part of the computation to compute nodes through slurm. This is not very useful for a pure python hands-on as the one on hdg 1d you have been working on Tuesday. But it might be of interest for you in the future to fully exploit plafrim.

    In this situation, you launch jupyter from the frontal node mistral01 exactly as above:

    guix environment --pure --ad-hoc jupyter python-ipython python-ipykernel python-matplotlib python-pandas python-scipy python-slurm-magic bash wget coreutils -- jupyter notebook --ip=127.0.0.1 --port=500<XY> --no-browser --NotebookApp.notebook_dir=/home/
    

    The server is thus running on the frontal node. The idea is to delegate (only) some of the computation (the costly ones) to the miriel comptute nodes through slurm. We use slurm-magic to do so. Note that in the above guix environment command line, the python-slurm-magic guix package is loaded to provide the service on the jupyter server side. So there is nothing more to do on the server side in our case.

    We are almost set to play on the client side (as above within your browser's laptop). Make a new cell (for instance at the top of your notebook, but not necessarily) and load the slurm-magic extension:

    %load_ext slurm_magic
    

    You can do view the available magic commands with (again type it in a jupyter cell and run it):

    %lsmagic
    

    Want to view the compute partitions available on plafrim?

    %sinfo
    

    We observe that the hpc partition onto which we view that miriel compute nodes are available. Let's run on that partition!

    %srun -p hpc /bin/hostname
    

    If you want to know more (%%sbatch, …), we invite you to a look at this more excellent tutorial.

  2. Running both the notebook server and the computation on a compute node (one more ssh bounce)
    hostname
    
    mistral01.formation.cluster
    

    The second option consists of reserving a compute node and performing an additional ssh bounce onto it.

    We reserve a compute node:

    salloc -p hpc
    
    salloc: Granted job allocation 426856
    

    We then check out the node we got allocated:

    squeue -u $USER
    
    JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
    426856 hpc bash hpcs-agu R INVALID 1 miriel032

    We then perform an additional bounce maintaining port redirection:

    ssh -L 500<XY>:localhost:500<XY> miriel0<ZZ>
    

    where <XY> and <ZZ> are to be once again changed (you personal digits and the miriel number – 32 – in our case, respectively).

    All is set up to start the notebook from the compute node

    guix environment --pure --ad-hoc jupyter python-ipython python-ipykernel python-matplotlib python-pandas python-scipy python-slurm-magic bash wget coreutils -- jupyter notebook --ip=127.0.0.1 --port=50034 --no-browser --NotebookApp.notebook_dir=/home/     
    

    You may copy and paste the proposed url into your laptop's browser. You may want to check out on which node you are running by typing and executing the following command in a cell of your notebook (the ! makes it a bash request instead of a python command):

    %srun -p hpc /bin/hostname
    

    The command has been run on a miriel node (miriel032 in our case).

2.5.2. Embedding reproducibility in the notebook

You may want to read this guix feature.

Author: Inria Bordeaux Sud Ouest

Created: 2021-06-10 Thu 18:01

Validate