HDG 2D-3D

Get connected to plafrim:

ssh plafrim-hpcs

We use the same guix environment as in the previous hands-on session.

Clone the hou10ni source code:

git clone https://gitlab.inria.fr/fuentes/hou10ni_school_2019.git
cd hou10ni_school_2019 
mkdir build
cd build

Set up the required guix environment and build hou10ni:

guix environment --pure maphys --ad-hoc maphys pastix starpu vim -- /bin/bash --norc
cmake ..
make

Note: If a package is not found (such as starpu which is provided by guix-hpc but not guix core), make sure you have enabled the channels with the set up we viewed yesterday). The guix describe command allows you to check whether the three channels you wanted to set up are effectively enabled. The guix pull -l command retrieves the list (the history) of the set up you have used so far.

You can test the installation with the following command:

ctest

You can now run on the frontal node (mistral01) as follows:

mpirun -np 2 ./hou10ni_lite.out < param_simple_mumps.txt 
mpirun -np 2 ./hou10ni_lite.out < param_simple_maphys.txt 

We now exit the guix environment

exit

All the runs in this hands-on session must be performed from within the build directory using the ./hou10ni_lite.out command (paths could be handled but not addressed in this hands-on).

We can now try to perform an interactive execution on a computing node (miriel in the case of plafrim-hpcs). Let's first check out whether there are partitions with available (idle) compute nodes:

sinfo

In this case, there are 19 miriel nodes available on the hpc partition. We reserve (salloc) 1 node (-N 1) on that partition (-p hpc):

salloc -p hpc -N 1

salloc: Granted job allocation 426500

We can monitor the node that we got with squeue (restricting the view to our username with the - u option):

squeue -u $USER

We can directly connect to that node (where <XXX> shall be 023 in this example and has to be adapted to the value you obtained):

ssh miriel<XXX>

Let's see whether we can reproduce the previous execution:

cd hou10ni_school_2019/build
guix environment --pure maphys --ad-hoc maphys pastix starpu vim -- /bin/bash --norc
mpirun -np 2 ./hou10ni_lite.out < param_simple_mumps.txt 
mpirun -np 2 ./hou10ni_lite.out < param_simple_maphys.txt 

You can monitor the CPU and memory usage by opening a new terminal on your laptop:

ssh plafrim-hpcs
ssh miriel<XXX>
top

In fortran, the time can be recorded thanks to:

CALL CPU_TIME(t1)

where t1 is real, of kind dp.

Hence, if you want to time some part of code, this can be done through:

CALL MPI_BARRIER(MPI_COMM_WORLD,ierr)
if (myrank.eq.0) then ! only on the master proc
CALL CPU_TIME(t1)
end if
! [...]
! part of the code to be timed
! [...]
CALL MPI_BARRIER(MPI_COMM_WORLD,ierr)
if (myrank.eq.0) then ! only on the master proc
CALL CPU_TIME(t2)
write(6,*) "Computation time for part 1 is ", t2-t1
end if

In the main program lib/bin/hou10ni_lite.F90 Evaluate the time spent for:

1. Initializing the problem (reading the data, reading and partionning the mesh)
2. Construction the global matrix
3. Solving the linear system

Check the scalability of the code by running param_simple_mumps.txt and param_simple_maphys.txt on 2, 4, 8, 16 and 24 cores on miriel.

To see only the relevant lines, you may use the command

mpirun -np 4 ./hou10ni_lite.out < param_simple_maphys.txt | grep "Computation time"

The lapack zgetrf double precision complex (z) general (ge) triangular (tr) factorization (f) routine is called by sub_defstiffmat_elastic_hybrid in lib/libmat/sub_defstiffmat_hybrid.F90. Propose and implement a way to evaluate the time spent in this routine for 2, 4 and 8 mpi threads.

Set the OMP_NUM_THREADS environment variable to 1 thanks to the command

export OMP_NUM_THREADS=1

Run the scalability tests once again.

In your maphys conf_maphys.in input file, turn on the dump of matrices by changing the line:

ICNTL(45)=1

Note that three files will be dumped per process (local matrix, local assembled right hand side and subdomain connectivity). The name of the files cannot be changed, so if you want to keep the matrices, create a new folder and put them inside.

For example, suppose you are working on a test case of your choice (numerical and computational set up) that you want to reference to as mymatofchoice1. After you have executed hou10ni in this set up with dump enabled, you can perform the following instructions:

mkdir maphys_mymatofchoice1
mv maphys_local_* maphys_mymatofchoice1/

In your houdini input file, turn on the dump of matrices by changing the line:

.FALSE. Do you want to output the matrices ? (.FALSE. or .TRUE.)

to:

.TRUE. Do you want to output the matrices ? (.FALSE. or .TRUE.)

With mumps, dumping the matrices can be done similarly. The matrix will be stored in files global_mat.txt<pid> (where pid is the process id) and the right and side will be stored in a centralized way in global_mat.txt.rhs.

mkdir mumps_mymatofchoice2
mv global_mat* mumps_mymatofchoice2/

Create a slurm batch file hou10ni.batch in the build directory

#!/bin/sh                                                                       
#SBATCH --time=03:59:00                                                         
#SBATCH -N 2                                                                    
#SBATCH -n 48                                                                   
#SBATCH -p hpc                                                                  
guix environment --pure maphys --ad-hoc maphys pastix starpu -- /bin/bash --norc
export OMP_NUM_THREADS=1
srun --mpi=pmi2  ./hou10ni_lite.out < param_simple_3D_maphys.txt

Submit the job

sbatch hou10ni.batch

You can check the state of the job by

squeue

When the job is running, you will see the nodes on which it is submitted and monitor the job by

ssh miriel
top

Have a look at the batch file above? Do you think ./hou10ni_lite.out is running in a guix environment?

If you have a doubt, check out what's going on with the following code:

guix environment --pure --ad-hoc openmpi openssh
mpirun -n 2 ./hou10ni_lite.out < param_simple_mumps.txt 

Does it give you a hint? Read the output of ldd ./hou10ni_lite.out outside of your guix environment. Got it?

You may want to further read the slurm/mpi/guix tutorial.

You may want to prepare batch to run large test cases on a larger number of nodes. For the largest ones, you may want to batch at the end of the session only in order to avoid consuming too many resources while everybody is working actively on the hands-on. Do not produce vtu or matrix outputs for very large test cases.