Morse + simgrid Hands-on

Table of Contents

1 Introduction

You can check out this tutorial and org-mode version here:

svn co svn+ssh://scm.gforge.inria.fr/svnroot/morse/tutorials/morse_simgrid

1.1 Prerequisites

  • a recent Emacs/Org-mode install (Emacs >=24.3 and Org-mode version >= 8.2 is recommended);
  • prerequisites for spack (module, etc: please refer to this tutorial)

1.2 Goal of this tutorial

We created a pad for notes, questions, bugs and suggestions concearning this tutorial. The goal of this tutorial is to explain, step by step, how to reproduce real execution and simulation of several solvers from MORSE. We consider here dense, sparse linear algebra libraries, like Chameleon, MUMPS, PaStiX, etc.

1.3 MORSE distribution via spack

spack-morse-packages.jpg

1.4 Emacs and Org-mode

This tutorial is written in Org-mode and it is based on its literate programming feature Org-babel. Although Org-mode files can be read in any text editors, we strongly encourage to use Emacs, as at this point only this editor allows for fully benefiting from Org-babel feature.

In order to run everything smoothly, you will need recent Emacs (>24.3) and a recent Org-mode (>8.0). If you have trouble installing this software, consult the following web page (or this one if you are a Mac user).

Once everything is installed, execute the following code by selecting it with cursor and applying the shortcut "Ctrl-c Ctrl-c" to test that everything works properly. This small code will execute shell command echo with the specified text.

#+NAME foo

   echo "foo"
foo

Now execute this code to set the environment in order to be able to execute shell and not have to confirm every command.

 (setq org-confirm-babel-evaluate nil)
 (setq org-export-babel-evaluate nil)
 (org-babel-do-load-languages 'org-babel-load-languages
      '(
       (sh . t)
       (python . t)
       (R . t)
       (org . t)
       ))
(setq org-support-shift-select t)

1.5 Configure your local working environment

First, prepare a clean directory to work in it. You should execute this section only once. If you don't want to use this directory, you're free to work in another directory, but you still need to set WORK_DIR variable.

mkdir -p morse_simgrid_tutorial
cd morse_simgrid_tutorial
export WORK_DIR=$PWD
cd -

2 Installing spack   @SIROCCO

If you have already installed spack (latest version from morse branch) and set correctly your environment, then you can skip this part. For morse information about spack, you can follow this link. You will find there informations about prerequisites and useful commands in spack.

Clone Spack git:

cd $WORK_DIR
git clone https://github.com/fpruvost/spack.git
cd spack
git checkout morse
git pull
export SPACK_ROOT=$WORK_DIR/spack
. $SPACK_ROOT/share/spack/setup-env.sh 
export MODULEPATH=$SPACK_ROOT/share/spack/modules/linux-x86_64:$MODULEPATH
mkdir -p $HOME/.spack
echo "Spack environment is set"

3 Install and check the MPI solver stack with MPICH

3.1 Install

Check the stacks to install:

spack spec pastix +mpi+examples
spack spec mumps +examples
spack spec scotch +grf+mpi ^mpich

Install the one you want to test:

spack install -v -n scotch +grf+mpi ^mpich
spack install -v -n pastix  +mpi+examples ^mpich
spack install -v -n mumps +mpi+examples ^mpich
spack install -v -n maphys@svn-maphys-dev +examples ^mpich

Source files are place in "/tmp/$USER", and are deleted when the installation is finished. You can keep source files by using the option –keep-stage, but they will still be kept in /tmp.

spack install -v -n --keep-stage scotch +grf+mpi ^mpich

Otherwise, you can use @src versions, which allows you to use source files you downloaded by yourself. For example, let's say you have scotch source installed in $HOME/scotch, you could install scotch using those sources and spack dependencies with:

export SCOTCH_DIR=$HOME/scotch
spack install -v -n scotch@src +grf+mpi ^mpich

3.2 Sanity checks

Test SCOTCH

spack load mpich
spack load scotch +grf+mpi ^mpich
export SCOTCH_PATH=`spack location -i scotch +grf+mpi ^mpich`
mpirun -np 3 dgpart 3 $SCOTCH_PATH/grf/3elt.grf

Test PaStiX

spack load mpich
spack load  pastix +mpi+examples   ^mpich
export PASTIX_PATH=`spack location -i pastix +mpi+examples  ^mpich`
mpirun -np 3 ${PASTIX_PATH}/examples/zsimple -mm $PASTIX_PATH/matrix/young4c.mtx

Test MUMPS

spack load mpich
spack load  mumps +mpi+examples ^mpich
export MUMPS_PATH=`spack location -i mumps +mpi+examples  ^mpich`
cd ${MUMPS_PATH}/examples/
mpirun ${MUMPS_PATH}/examples/c_example

Test MaPHyS

spack load mpich
spack load maphys@svn-maphys-dev +examples ^mpich
export MAPHYS_PATH=`spack location -i maphys@svn-maphys-dev +examples  ^mpich`
cd ${MAPHYS_PATH}/examples/
mpirun -np 4 ${MAPHYS_PATH}/examples/dmph_examplekv real_bcsstk17.in

4 Install and check the MPI solver stack with SMPI

BEWARE: This section is not yet functional.

4.1 Install

spack install -v -n scotch +grf+mpi ^simgrid@master+examples
spack install -v -n pastix +mpi+examples ^simgrid@master+examples
spack install -v -n mumps +mpi+examples ^simgrid@master+examples
spack install -v -n maphys@svn-maphys-dev +examples ^simgrid@master+examples

4.2 Sanity checks

Test SCOTCH

spack load simgrid@master+examples
spack load scotch +grf+mpi ^simgrid@master+examples
export SCOTCH_PATH=`spack location -i scotch +grf+mpi ^simgrid@master+examples`
export SIMGRID_PATH=`spack location -i simgrid@master+examples`
smpirun -trace -trace-resource -trace-viva -trace-file smpi_traced.trace -hostfile $SIMGRID_PATH/examples/smpi/hostfile -platform $SIMGRID_PATH/examples/smpi/../platforms/small_platform.xml --cfg=path:$SIMGRID_PATH/examples/smpi/../msg --cfg=smpi/cpu_threshold:-1 -np 3 --log=smpi_kernel.thres:warning dgpart 3 $SCOTCH_PATH/grf/3elt.grf
afalco@afalco-hp:~/tutorials/morse_simgrid$ afalco@afalco-hp:~/tutorials/morse_simgrid$ afalco@afalco-hp:~/tutorials/morse_simgrid$ afalco@afalco-hp:~/tutorials/morse_simgrid$ [0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing' to 'yes'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing/filename' to 'smpi_traced.trace'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing/smpi' to 'yes'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing/categorized' to 'yes'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing/uncategorized' to 'yes'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'viva/categorized' to 'smpi_cat.plist'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'viva/uncategorized' to 'smpi_uncat.plist'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'surf/precision' to '1e-9'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'network/model' to 'SMPI'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'network/TCP_gamma' to '4194304'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/cpu_threshold' to '-1'
dgpart(2): ERROR: main: too many file names given
[Fafard:2:(3) 0.000000] [instr_config/INFO] No categories declared, ignoring generation of viva graph configuration
[Fafard:2:(3) 0.000000] /tmp/afalco/spack-stage/spack-stage-ZOMJ2N/simgrid/src/simix/smx_global.cpp:281: [simix_kernel/CRITICAL]    
[Fafard:2:(3) 0.000000] /tmp/afalco/spack-stage/spack-stage-ZOMJ2N/simgrid/src/simix/smx_global.cpp:282: [simix_kernel/CRITICAL] The time is still 0, and you still have processes ready to run.
[Fafard:2:(3) 0.000000] /tmp/afalco/spack-stage/spack-stage-ZOMJ2N/simgrid/src/simix/smx_global.cpp:283: [simix_kernel/CRITICAL] It seems that you forgot to run the simulation that you setup.
[Fafard:2:(3) 0.000000] /tmp/afalco/spack-stage/spack-stage-ZOMJ2N/simgrid/src/simix/smx_global.cpp:284: [xbt/CRITICAL] Bailing out to avoid that stop-before-start madness. Please fix your code.
dgpart(0): ERROR: main: unprocessed option '–cfg=tracing:yes'

Test PaStiX

spack load simgrid@master+examples
spack load  pastix +mpi+examples ^simgrid@master+examples
export PASTIX_PATH=`spack location -i pastix +mpi+examples ^simgrid@master+examples`
export SIMGRID_PATH=`spack location -i simgrid@master+examples`
smpirun -trace -trace-resource -trace-viva -trace-file smpi_traced.trace -hostfile $SIMGRID_PATH/examples/smpi/hostfile -platform $SIMGRID_PATH/examples/smpi/../platforms/small_platform.xml --cfg=path:$SIMGRID_PATH/examples/smpi/../msg --cfg=smpi/cpu_threshold:-1 -np 3 --log=smpi_kernel.thres:warning ${PASTIX_PATH}/examples/zsimple -mm $PASTIX_PATH/matrix/young4c.mtx
afalco@afalco-hp:~/tutorials/morse_simgrid$ afalco@afalco-hp:~/tutorials/morse_simgrid$ afalco@afalco-hp:~/tutorials/morse_simgrid$ [0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing' to 'yes'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing/filename' to 'smpi_traced.trace'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing/smpi' to 'yes'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing/categorized' to 'yes'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing/uncategorized' to 'yes'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'viva/categorized' to 'smpi_cat.plist'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'viva/uncategorized' to 'smpi_uncat.plist'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'surf/precision' to '1e-9'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'network/model' to 'SMPI'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'network/TCP_gamma' to '4194304'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/cpu_threshold' to '-1'
MPI_Init_thread level = MPI_THREAD_SINGLE
driver: MatrixMarket file: /home/afalco/spack/opt/spack/linux-x86_64/gcc-4.8/pastix-5.2.2.22-iic3ydcydxz3saz73wunrc7phagabhzi/matrix/young4c.mtx
open RHS file : /home/afalco/spack/opt/spack/linux-x86_64/gcc-4.8/pastix-5.2.2.22-iic3ydcydxz3saz73wunrc7phagabhzi/matrix/young4c.mtx.rhs
cannot load /home/afalco/spack/opt/spack/linux-x86_64/gcc-4.8/pastix-5.2.2.22-iic3ydcydxz3saz73wunrc7phagabhzi/matrix/young4c.mtx.rhs
Setting right-hand-side member such as X[i] = i + i*I
Check : Numbering		OK
Check : Sort CSC		OK
Check : Duplicates		OK
 +--------------------------------------------------------------------+
 +              PaStiX : Parallel Sparse matriX package               +
 +--------------------------------------------------------------------+
  Matrix size                                   841 x 841
  Number of nonzeros in A                       2465
 +--------------------------------------------------------------------+
 +  Options                                                           +
 +--------------------------------------------------------------------+
	Version             :                   5.2.2.22
	SMP_SOPALIN         :                   Defined
	VERSION MPI         :                   Defined
	PASTIX_DYNSCHED     :                   Not defined
	STATS_SOPALIN       :                   Not defined
	NAPA_SOPALIN        :                   Defined
	TEST_IRECV          :                   Not defined
	TEST_ISEND          :                   Defined
	TAG                 :                   Exact Thread
	FORCE_CONSO         :                   Not defined
	RECV_FANIN_OR_BLOCK :                   Not defined
	OUT_OF_CORE         :                   Not defined
	DISTRIBUTED         :                   Not defined
	METIS               :                   Not defined
	WITH_SCOTCH         :                   Defined
	INTEGER TYPE        :                   int
	PASTIX_FLOAT TYPE   :                   double complex
 +--------------------------------------------------------------------+
   Time to compute ordering                     0.00783 s
   Time to analyze                              0.000149 s
   Number of nonzeros in factorized matrix      21756
   Fill-in                                      8.82596
   Number of operations (LLt)                   3.14977e+06
   Prediction Time to factorize (AMD 6180  MKL) 0.000389 s
   --- Sopalin : Threads are binded                 ---
[0.092981] [instr_config/INFO] No categories declared, ignoring generation of viva graph configuration
[0.092981] /tmp/afalco/spack-stage/spack-stage-ZOMJ2N/simgrid/src/simix/smx_global.cpp:546: [simix_kernel/CRITICAL] Oops ! Deadlock or code not perfectly clean.
[0.092981] [simix_kernel/INFO] 3 processes are still running, waiting for something.
(<name>@<host>): <status>"
[0.092981] [simix_kernel/INFO] Process 1 (0@Tremblay): waiting for communication synchro 0x1d7bcc0 ((null)) in state 0 to finish
[0.092981] [simix_kernel/INFO] Process 2 (1@Jupiter): waiting for communication synchro 0x1d7b810 ((null)) in state 0 to finish
[0.092981] [simix_kernel/INFO] Process 3 (2@Fafard): waiting for communication synchro 0x1d7bae0 ((null)) in state 0 to finish

Test MUMPS

spack load simgrid@master+examples
spack load  mumps +mpi+examples ^simgrid@master+examples
export MUMPS_PATH=`spack location -i mumps +mpi+examples  ^simgrid@master+examples`
cd ${MUMPS_PATH}/examples/
export SIMGRID_PATH=`spack location -i simgrid@master+examples`
smpirun -trace -trace-resource -trace-viva -trace-file smpi_traced.trace -hostfile $SIMGRID_PATH/examples/smpi/hostfile -platform $SIMGRID_PATH/examples/smpi/../platforms/small_platform.xml --cfg=path:$SIMGRID_PATH/examples/smpi/../msg --cfg=smpi/cpu_threshold:-1 -np 3 --log=smpi_kernel.thres:warning ${MUMPS_PATH}/examples/c_example
PB allocation in DMUMPS_LOAD_INIT

Test MaPHyS

spack load simgrid@master+examples
spack load maphys@svn-maphys-dev +examples ^simgrid@master+examples
export MAPHYS_PATH=`spack location -i maphys@svn-maphys-dev +examples  ^simgrid@master+examples`
cd ${MAPHYS_PATH}/examples/
export SIMGRID_PATH=`spack location -i simgrid@master+examples`
smpirun -trace-resource -trace-viva -trace-file smpi_traced.trace -hostfile $SIMGRID_PATH/examples/smpi/hostfile -platform $SIMGRID_PATH/examples/smpi/../platforms/small_platform.xml --cfg=path:$SIMGRID_PATH/examples/smpi/../msg --cfg=smpi/cpu_threshold:-1 -np 4 --log=smpi_kernel.thres:warning ${MAPHYS_PATH}/examples/dmph_examplekv real_bcsstk17.in
afalco@afalco-hp:~/spack/opt/spack/linux-x86_64/gcc-4.8/mumps-5.0.1-r7drhifwv4ll3umvxumilaezdlgc6y44/examples$ afalco@afalco-hp:~/spack/opt/spack/linux-x86_64/gcc-4.8/mumps-5.0.1-r7drhifwv4ll3umvxumilaezdlgc6y44/examples$ afalco@afalco-hp:~/spack/opt/spack/linux-x86_64/gcc-4.8/maphys-svn-maphys-dev-hzs2mv5xodqvf3b2qjw22krvtailqhcv/examples$ afalco@afalco-hp:~/spack/opt/spack/linux-x86_64/gcc-4.8/maphys-svn-maphys-dev-hzs2mv5xodqvf3b2qjw22krvtailqhcv/examples$ [0.000000] [xbt_cfg/INFO] Configuration change: Set 'surf/precision' to '1e-9'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'network/model' to 'SMPI'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'network/TCP_gamma' to '4194304'
[0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/cpu_threshold' to '-1'
** SimGrid: UNCAUGHT EXCEPTION received on /home/afalco/spack/opt/spack/linux-x86_64/gcc-4.8/maphys-svn-maphys-dev-hzs2mv5xodqvf3b2qjw22krvtailqhcv/examples/dmph_examplekv(1): category: network error; value: 1
** Link failure
** Thrown by maestro() on process 0
[Tremblay:0:(1) 0.029808] /tmp/afalco/spack-stage/spack-stage-ZOMJ2N/simgrid/src/xbt/ex.c:149: [xbt_ex/CRITICAL] Link failure

**   In SIMIX_comm_finish() at /tmp/afalco/spack-stage/spack-stage-ZOMJ2N/simgrid/src/simix/smx_network.cpp:809
**   In SIMIX_post_comm() at /tmp/afalco/spack-stage/spack-stage-ZOMJ2N/simgrid/src/simix/smx_network.cpp:932
**   In SIMIX_run() at /tmp/afalco/spack-stage/spack-stage-ZOMJ2N/simgrid/src/simix/smx_global.cpp:472
**   In smpi_main() at /tmp/afalco/spack-stage/spack-stage-ZOMJ2N/simgrid/src/smpi/smpi_global.c:693
**   In __libc_start_main() at /build/buildd/eglibc-2.19/csu/libc-start.c:321
**   In _start() at ??:?

Author: HiePACS/STORM

Created: 2016-02-02 Tue 11:54

Validate