Swiss National Supercomputing Centre: Hints for using Mvapich

Hints for using Mvapich (V1.1.X) on rigi.cscs.ch

Foreground

This Makefile and examples is based on few sample code (in C, CC, F90) kindly provided by HP for testing the basic setup of the MPI environment. They have been slightly adapted for being executed on rigi.

Example source code

The examples described in this document can be downloaded (MvapichExample.tgz).

Mvapich Execution Environment Setup

[for tcsh users] consider adding in $HOME/.cshrc

  % source /etc/profile.d/modules.csh
  % module unload mpi
  % module load   mvapich

[for bash users] consider adding in $HOME/.bashrc

  % source /etc/profile.d/modules.bash
  % module unload mpi
  % module load   mvapich

Check that mvapich/xx.xx this is now the default environment:

  % ssh rigi        module list
  % ssh compute-0-1 module list

Mvapich Compiling Environment

Please make sure a correct pgi and gcc compiling env. is loaded (i.e):

  % pgf90 -V
  pgf90 7.1-6 64-bit target on x86-64 Linux -tp k8-64e

  % gcc --version
  gcc (GCC) 3.4.6 20060404 (Red Hat 3.4.6-9)

  NOTE: (GCC) 4.X should also work

Check compiling and linking wrappers

  % mpicc  --version
  gcc (GCC) 3.4.6 20060404 (Red Hat 3.4.6-9)

  % mpiCC  --version
  g++ (GCC) 3.4.6 20060404 (Red Hat 3.4.6-9)

  % mpif90 -V
  pgf90 7.1-6 64-bit target on x86-64 Linux -tp k8-64e

  % mpif77 -V
  pgf90 7.1-6 64-bit target on x86-64 Linux -tp k8-64e

Compiling and Linking the Examples

  % make clean
  % make

Interactive Execution

Discouraged and even forbidden: however in this case is useful for ensuring the mpi execution environment is correctly setup.

  % mpirun -np 8 -hostfile pbs_nodefile8.test ./hello_world

Batch Job Example (PBS)

job_example.sh should be slightly adapted for your environment.

job_example.sh

  #!/bin/bash
  #PBS -N Mvapich-example
  #PBS -m abe
  #PBS -M Your_Email@Address
  #PBS -l select=4:ncpus=1:mpiprocs=1:mem=512mb
  ##PBS -l place=scatter
  ##PBS -l place=scatter:excl
  #PBS -l walltime=00:30:00
  #PBS -q feed@rigi.cscs.ch
  #PBS -r n
  #======START=====
  echo start

  WORKDIR="$HOME/www-examples/MvapicExample"
  echo "- WORKDIR $WORKDIR"
  cd $WORKDIR

  echo "- Current PBS_NODEFILE:"
  echo $PBS_NODEFILE
  cat  $PBS_NODEFILE

  echo "- which mpirun:"
  which mpirun

  CMD="mpirun -np 4 -hostfile $PBS_NODEFILE $WORKDIR/hello_world"
  echo "- Executing: $CMD"
  $CMD

  echo stop
  #=====END=====

Submitting the job and checking the job status:

  % make hello_world
  % qsub job_example.sh
  % qstat -a
  % qstat -a -u $USER

Checking job results:

  % cat Mvapich-example.o<Job#>
  ...
  start
  - WORKDIR /users/xxx/www-examples/MvapicExample
  - Current PBS_NODEFILE:
  /var/spool/PBS/aux/10457.rigi.cscs.ch
  compute-0-1
  compute-0-3
  compute-0-5
  compute-0-5
  - which mpirun:
  /apps/mvapich-1.1rc1/bin/mpirun
  - Executing: mpirun -np 4 -hostfile /var/spool/PBS/aux/10457.rigi.cscs.ch /users/xxx/www-examples/MvapicExample/hello_world
  Hello world! I'm 0 of 4 on compute-0-1.local
  Hello world! I'm 1 of 4 on compute-0-3.local
  Hello world! I'm 2 of 4 on compute-0-5.local
  Hello world! I'm 3 of 4 on compute-0-5.local
  stop
  === BEGIN EPILOGUE ===
  ...

  % cat Mvapich-example.e<Job#>
    <should be empty>