EN Bereichsnavigation EN

event list

Registration

Sorry, the registration period for this event is over.

Fast Parallel I/O on HPC systems - 9-10 June 2011

 

Improved scalability of parallel computational kernels in modern HPC applications frequently exposes the time taken to access data storage devices as the major continuing bottleneck to further scalability increases. Many applications serialize data output through a single processor, and this limits their ability to take advantage of the full capabilities of the fast storage systems attached to modern HPC machines. This course looks at the capabilities of modern HPC fast storage infrastructure, the filesystems that are designed to make efficient use of this hardware and the most frequently used parallel libraries that allow applications to spread the I/O load across multiple processes and provide good read and write performance on these systems.

Registration deadline: June 7, 2011. 

Please contact nstring(at)cscs.ch for further technical information and apinna(at)cscs.ch for logistical information.

Instructors

Matthew Cordery, Jean Favre, Neil Stringfellow, CSCS

Venue

CSCS, Via Cantonale, Galleria 2, 6928 Manno  (Please note that on googlemap CSCS is wrongly posted)

www.cscs.ch/about_us/visitor_information/index.html

Time

13:00 - 17:00 day 1; 09:00 - 17:00 day 2

Target system

The material in this course will be applicable to a wide range of HPC architectures, with some specific material tailored to CSCS production Cray systems «Monte Rosa» and «Piz Palu»

Prerequisites

Participants are expected to bring a laptop for hands-on training.

Login onto Palu will be via ssh using wlan

Maximum number of participants

28

Accommodation

Participants are kindly requested to make their own arrangements for accommodation

Course Agenda

During this two day course we will look at the following topics:

Filesystems
==========
1) Parallel filesystems

  •     Essential hardware components - storage medium, controllers, network infrastructure and file servers
  •     Common parallel filesystems - Lustre, GPFS
  •     Basic common filesystem features
  •     Metadata and data - separation of metadata and data
  •     Multiple data servers and resiliency
  •     Filesystem bottlenecks

2) Filesystems on supercomputers
    Types of filesystems that are normally available

  •     Contention issues with shared filesystems
  •     Contention for the filesystem
  •     Contention on the filesystem
  •     Contention on the network
  •     Contention for metadata operations

3) Best practices when using filesystems

  •     Target users of parallel filesystems
  •     Performance issues when using multiple directories and files
  •     Responsible usage and how one user can affect others on the same filesystem


Parallel I/O
========
1) Selection of portable data formats

  •     Data storage considerations
  •     Data provenance
  •     Raw binary data and formatted data
  •     Metadata
  •     Restart files and data analysis files
  •     Popular portable data formats
  •     Tools support for data formats

2) Common application strategies for outputting data

  •     All writing through process zero
  •     All processes write to individual files
  •     Use of a subset of nodes for aggregated data
  •     Full collective output


3) Issues to consider for parallel input

4) Parallel I/O libraries

  •     MPI-2 parallel I/O

                 Using MPI-IO
                 Collective and independent buffering
                 MPI subarray types for collective I/O
                 Use of MPI hints to improve I/O access
                 Cray XT/XE MPI-IO on Lustre file systems

  •    HDF5

                HDF5 as a portable data format
                HDF5 parallel built on top of MPI-IO
                HDF5 groups, datasets and datatypes
                HDF5 hyperslabs
                HDF5 performance
                Post-processing HDF5 data files

  •    NetCDF

                NetCDF as a portable data format
                NetCDF4 parallel built on top of HDF5 parallel
                NetCDF array concepts
                Performance considerations when using NetCDF-4 parallel in applications
                Post-processing NetCDF data files

  •   Other portable data libraries and formats

               ADIOS and DataStager
               SIONlib

The course will be a mix of lectures, presentations, and hands-on sessions. 



Back to listing