The following instructions are for those who want to get Totalview up and running quickly.

• ssh -Y rosa.cscs.ch
• ftn -g -o myprogram myprogram.f90
• module load xt-totalview
• qsub -I -V -l mppwidth=12,walltime=00:15:00
• totalview aprun -a -n 12 ./myprogram

Read on for complete details.

To top

TotalView is a debugger with support for Fortran, C, C++, MPI, OpenMP and threads.

                   TotalView is an interactive tool that lets you debug serial, multiprocessor and multithreaded programs. It can be executed either as a graphical user interface (by using the totalview executable) or from a command-line interface (by using the totalviewcli executable). Totalview provides source-level debugging of Fortran and Fortran 90, C, and C++ codes. It can be used to debug parallel programs based on MPI. It also has facilities for multi-process thread-based parallel programs such as OpenMP.

CSCS characteristics

• Totalview is currently Installed on all the CRAY XT machines at CSCS. In order to check which version is available, type

                                      module avail xt-totalview

• There are no interactive nodes reserved for debugging. You will need to launch TotalView from within an interactive session.
• Note that the interactive sessions are strictly meant for debugging, running anything else on these compute nodes is prohibited.
• CSCS current license allows for 2 concurrent TotalView users on Rosa or Palu. Two of those users can use up to 128 cores. Currently, 8 users may also run Totalview interactively using up to 8 processors on Gele.
• If larger debugging runs are needed, please contact CSCS Help.

Totalview Limitations

Cray XT5 supports a special implementation of the Etnus Totalview debugger. The Totalview debugging suite for the Cray XT systems differs in functionality from the standard Totalview implementation in the following ways :

• Debugging multiple threads on compute nodes is not supported.
• Debugging MPI_Spawn(), OpenMP, Cray SHMEM, or PVM programs is not supported.
• Compiled EVAL points and expressions are not supported.
• Type transformations for the PGI C++ compiler standard template library collection classes are not supported.
• Exception handling for the PGI C++ compiler runtime library is not supported.
• Spawning a process onto the compute processors is not supported.
• Machine partitioning schemes, gang scheduling, or batch systems are not supported.
• The Totalview Visualizer is not included. The Totalview HyperHelp browser is not included.

For details please have a look at the documentation.

To top

Requirements

• For source level debugging, the program must be compiled with the flag -g :

                ftn -g -o myprogram myprogram.f90

• You need to turn on X-forwarding in order to be able to use the GUI of Totalview, which is done by adding the -Y or -X flag when you login via ssh (run "man ssh" for an explanation of the flags) :

                ssh -Y rosa.cscs.ch

Usage

Example1 : Invoking Totalview to debug a MPI application

• First, load the totalview module to get the correct environment variables set :

                module load xt-totalview

• If you want to use TotalView to start your job and monitor it as it runs, you must take some additional steps. Because parallel jobs cannot be launched directly from the login nodes, you will need to launch TotalView from within a batch job. The easiest way to do this is to start an interactive batch job as follows:

                qsub -I -V -l mppwidth=12,walltime=00:15:00

• Make sure to use -V as an option to qsub because it will import your current environment (including the DISPLAY variable for the X-Window connection). Once your job starts, you can launch TotalView graphical user interface from the command line. The following example starts TotalView on 12 compute cores:

                totalview aprun -a -n 12 ./myprogram

• The normal aprun options are given after the TotalView flag -a.  Starting Totalview will make two windows appear on the screen. These are called the Totalview Root and Process Window (or the main window).
• To start the execution click the Go button on the main window.
• This results in TotalView asking if the user wants to stop the program, e.g., for inserting break points. Click YES and the source code for the main program is displayed on the main window.
• Insert a breakpoint. Source lines where it is possible to insert a breakpoint are marked with a box in the left column. Click on a box to toggle a breakpoint.
• The execution can then be started with the Go button. The button "Go" runs the program from the beginning until the first breakpoint. "Next" and "Step" takes you one line forward. "Out" will continue until the end of the current subroutine/function. "Run to" will continue until the next breakpoint.
• The value of variables can be inspected by right clicking on the name, then choose "add to expression list". The variable will now be shown in a pop up window. Scalar variables will be shown with their value, arrays with their dimensions and type. To see all values in the array, right click on the variable in the pop up window and choose "dive". You can now scroll through the list of values.
• Another option (very useful) is to visualize the array : after choosing "dive", open the menu item "Tools->Visualize" of the pop up window. If you did this with a 2D array, use middle button and drag mouse to rotate the surface that popped up, shift+middle button to pan, Ctrl+middle button to zoom in/out.

For more information, see Troubleshooting below.

To top

• This page describes how to launch TotalView on the CRAY XT. For further information please check the TotalView documentation.

• In some cases, Totalview functionality is limited because Compute Node Linux (CNL) does not support some features in the user program. Please have a look at the Cray XT Programming Environment User's Guide (Chapter 11)

• Once inside the debugger, if you cannot see any source code, and if you keep the source files in a separate directory, add the search path to the src directory via the main menu item File->Search path.
• Check your ~/.totalview/preferences6.tvd preferences file.
• If everything fails, contact the helpdesk

• If you get the following error : 

  INFO: Copying library "/dsl/var/spool/alps/458302" into the local
  file cache ...
  ERROR: load_library_into_cache: Error closing cache file; errno=Disk
  quota exceeded(122)

  please clean your ~/.totalview/lib_cache directory.

 

To top

• The use of Totalview's GUI can sometimes be slow when you login via ssh. As a workaround, the TotalView Remote Display Client will let you launch TotalView on a remote system. Starting the Remote Display Client on your system will display a window into which you will be able to enter information about how the Remote Display can go from your system to the system upon which TotalView will execute. As Remote Display invokes TotalView on the remote host, it does not need be installed on your local machine. The Client can run on Linux x86, Linux x86-64, Darwin and Windows systems. No license is needed to run the Client.

Usage

Example : Invoking remote display from a Linux workstation

• First, choose and download the client corresponding to your operating system :

                uname -a    ( x86_64 GNU/Linux )

                scp -C  username@ela.cscs.ch:/opt/totalview/xt-totalview/remote_display/remote_display.1.1.0-0-linux-x86-64.tar .

• Install the client on your workstation and set the correct environment variables :

                tar xf remote_display.1.1.0-0-linux-x86-64.tar

                export LD_LIBRARY_PATH=~/remote_display/remote_display.1.1.0-0/linux-x86-64/shlib:$LD_LIBRARY_PATH

• Start the client :

                cd remote_display.1.1.0-0/linux-x86-64/bin

                ./remote_display_client

 

To top

The modulefile, xt-totalview-mem-debug, sets up the compilation so that the proper libraries will be linked for TotalView memory debugging. Load this modulefile only when memory debugging is desired.

module load xt-totalview/8.8.0a
module list -t
   totalview-support/1.1.1
   xt-totalview/8.8.0a

module load xt-totalview-mem-debug

please add -Wl,-zmuldefs to your compilation line

ftn -g -Wl,-zmuldefs mem_bug.f90 -o mem_bug.exe
/usr/bin/ld: Warning: size of symbol `free' changed from 14 in /opt/toolworks/totalview.8.8.0a/linux-x86-64/lib/libtvheap_cnl_static.a to 154 in /usr/lib64/libc.a(malloc.o)
/usr/bin/ld: Warning: size of symbol `malloc' changed from 14 in /opt/toolworks/totalview.8.8.0a/linux-x86-64/lib/libtvheap_cnl_static.a to 478 in /usr/lib64/libc.a(malloc.o)
/usr/bin/ld: Warning: size of symbol `realloc' changed from 14 in /opt/toolworks/totalview.8.8.0a/linux-x86-64/lib/libtvheap_cnl_static.a to 549 in /usr/lib64/libc.a(malloc.o)

To top