June 17, 2021 – by CSCS
As the largest research institution for natural and engineering sciences in Switzerland, the Paul Scherrer Institute (PSI) generates petabytes of data per year with experiments at research facilities such as the Swiss Light Source (SLS), the Swiss X-ray Free Electron Laser (SwissFEL) and Swiss Spallation Neutron Source (SINQ). PSI has been storing several petabytes of this data on a highly scalable storage solution at the Swiss National Supercomputing Centre (CSCS) since 2018. From 2024 onwards, however, PSI predicts exponential data growth — several hundred petabytes of data are expected to be generated, in fact. This has forced researchers, computer scientists and software engineers to rethink data workflow between the institutions and how data is accessed, processed, analysed and stored.
The PSI-CSCS joint project SELVEDAS (Services for Large Volume Experiment-Data Analysis utilising Supercomputing and Cloud technologies) offers a successful approach to handling the challenge. The project was launched in 2019 with the help of start-up funding from a swissuniversities grant. The grant was awarded to implement three data focused use cases, identified by PSI scientists. SELVEDAS's ambition was to provide CSCS computing and data services in a cloud infrastructure and to demonstrate the feasibility and benefits. The project successfully concluded in March 2021.
A dedicated network connection was established between PSI and CSCS to ensure secure, direct and high-speed data transfers between the two institutions for access, processing, analysis and storage of the data. This network connection was further upgraded from originally two 10-gigabyte bandwidths to two 100-gigabyte bandwidths to accommodate the expected growth of data. SELVEDAS should enable workflows on this infrastructure for extreme data volumes in near real time.
Data processing during ongoing experiments
One of the most challenging use cases of the project was to enable a data workflow where PSI scientists conducting experiments on one of the research facilities at PSI could transfer data immediately to CSCS, leveraging on the dedicated network connection. For example, when researchers study the internal nano structures of objects with the Swiss Light Source, the results must be immediately processed and returned to PSI, as the next step of the experiment is dependent on the processed form of the data.
As both PSI and CSCS serve different clients, it is highly complex to integrate such a use case, as both institutions are expected to maintain their autonomy to meet their internal and external requirements. The final design takes these constraints into consideration and successfully demonstrates a solution that can be applied across institutional and physical boundaries, with respect to customer requirements and their respective political restrictions.
To bring this design to life, CSCS developed Sarus and FirecREST as core components. Sarus enables scientists to run their portable scientific workflows in portable containers while ensuring native performance on HPC systems. FirecREST provides a ubiquitous interface to support complex scientific workflows while facilitating institutional needs to maintain their respective autonomy.
Pathfinders for the future
The PSI-CSCS team is convinced that collaborative efforts such as SELVEDAS are pathfinders for identifying future, scalable data infrastructure and solutions. They are a blueprint for future collaborations under "Alps", the new supercomputer infrastructure being developed at CSCS. "Alps", will provide users with a software-defined research infrastructure that uses cloud methodology for multitenancy, supercomputing, and data infrastructures, and where PSI can be a tenant. "With SELVEDAS, we have achieved an important milestone in showing the potential of cloud-based HPC services to PSI and other potential CSCS collaborators", says CSCS’s SELVEDAS project leader Cerlane Leong. Alun Ashton, Leong’s counterpart at PSI, points out: "Through the successful collaboration, CSCS has demonstrated how "Alps" could offer a future software-defined research infrastructure to PSI."
Image above: Swiss Light Source, PSI (Image: Schölla Schwarz)
- Leong SH, Stadler HC, Chang MC, Dorsch JP, Aliaga T and Ashton AW SELVEDAS: A data and compute as a service workflow demonstrator targeting supercomputing ecosystems, In 2020 IEEE/ACM International Workshop on Interoperability of Supercomputing and Cloud Technologies (SuperCompCloud) (2020), pp. 7-13, IEEE.
- Klein M, Martinasso M, Leong SH and Alam SR: Interactive supercomputing for experimental data-driven workflows (2019), In Tools and Techniques for High Performance Computing (2019), pp. 164-178. Springer, Cham.
- Benedicic L, Cruz FA, Madonna A, & Mariotti K: Sarus: Highly scalable docker containers for hpc systems, In International Conference on High Performance Computing (2019), pp. 46-60, Springer, Cham.
- Cruz FA & Martinasso M: Firecrest: Restful api on cray xc systems. arXiv preprint arXiv:1911.13160(2019).
- Alps research infrastructure: https://www.cscs.ch/science/computer-science-hpc/2021/cscs-director-shares-details-about-planned-successor-to-piz-daint/
- Sarus and FirecREST: https://products.cscs.ch