September 04, 2023 - by Simone Ulmer
The effects of climate change are contributing to more frequent extreme weather events, such as sustained high temperatures, destructive thunderstorms, and torrential rainfall, or sometimes even hailstones the size of a fist. Identifying and accurately predicting such extreme events in weather forecasts is both societally and economically relevant. However, classical formulations, so-called hydrostatic models, are often too coarse for this task. For example, they are not able to map the details of thunderstorm cells, because they can only calculate weather forecasts with a spatial grid spacing of several kilometres or more.
For more precise forecasts of extreme weather events, the models must calculate small-scale physical processes with even higher resolution. For example, for central Europe, models need to account for how air moves vertically within thunderstorms and interacts with the complex orography of the Alps, i.e. the topographical arrangement with steep slopes. “Here we need so-called non-hydrostatic models with grid spacing of one kilometre or less,” explains Heini Wernli, professor of atmospheric dynamics at ETH Zurich. “This is the only way we can represent the acceleration of air parcels in thunderstorms and near mountains as a result of the non-linear interaction of various processes."
Less than one hour for the 10-day forecast
For such complex calculations on a kilometre and sub-kilometre scale, the models must be better implemented on modern computer architectures. That means they need to be adapted to the architecture in a way that better utilises the supercomputer’s capabilities in order to calculate the forecasts cost-efficiently and significantly faster than in real time.
"The current world-leading model is an absolute success story," says Wernli, referring to the Integrated Forecasting System (IFS) introduced in the 1980s at the European Centre for Medium-Range Weather Forecasts (ECMWF). It was programmed in the computer language Fortran and has been continuously adapted and improved to keep up with the latest developments in research and technology. The model solves the hydrostatic equations with a so-called spectral method and operates highly efficiently on CPU-based computers. "The efficiency of the model is crucial, because we only have a short time window of less than one hour to calculate our global medium-range forecasts," says Christian Kühnlein, Senior Scientist at ECMWF.
But, in addition to further developing the current IFS, ECMWF has also been working for the last years on the FVM model, which solves the non-hydrostatic equations for the high resolutions with a finite-volume approach, as this method scales better on modern computer architectures than the spectral method. Now, the Swiss National Supercomputing Centre (CSCS), ETH Zurich, and the ECMWF, under the leadership of Kühnlein and Wernli, are joining forces in the PASC project KILOS to fundamentally revise the implementation of the FVM model on modern computer architectures. KILOS stands for "Kilometer-scale non-hydrostatic global weather forecasting with IFS-FVM", and the aim of the project is to make FVM highly efficient on the latest computer architectures with graphics processing units (GPUs). To this end, the FVM dynamical core and the central parameterisations of the IFS are being rewritten in the Python programming language with the so-called GT4Py library so that, one day, global weather forecasts can be calculated with a grid spacing of one kilometre or less. "GT4Py is a domain-specific Python library for weather and climate models that is developed by CSCS and ETH Zurich," says Kühnlein.
Animation of surface wind in the Schwende valley of Appenzell (Switzerland) during a Laseyer windstorm, as simulated in high resolution with the new FVM model. The gray contours show the height of the topography, and the green crosses mark the location of the two measurement stations in the region. (Animation: The KILOS research team)
Separating domain science and hardware aspects
During the transfers of the original Fortran programming of the FVM into the Python programming language, the KILOS team inserts certain instructions so that the toolchain of the GT4Py library, part of the GridTools Framework, implements and optimises the code on the available hardware. GT4Py separates the code components dedicated to the physical-mathematical model from those needed for optimal hardware use. "This makes it possible to develop and maintain the model more productively," says Kühnlein. More precisely, a hardware-specific programming language is automatically generated according to GT4Py’s instructions that ensures that the system is utilised efficiently, for example the programming language CUDA is used on GPUs made by NVIDIA. These GPUs will be installed in the new flagship system of CSCS, "Alps", which will replace "Piz Daint" in 2024. "Besides NVIDIA GPUs, however, there are others, such as those from AMD. GT4Py can generate specific hardware-optimised codes for different architectures," says Kühnlein. This efficient portability of the model on different GPUs is an overarching goal of the project.
According to the researchers, the newly formulated numerical equations in the dynamical core should also expand the capabilities of the model. "The dynamical core describes and calculates purely the fluid dynamics in the atmosphere, for example how the three-dimensional wind field develops as a function of time," says Wernli. Many physical phenomena outside of fluid mechanics, on the other hand, are described via parameterisations: If, for example, the flow causes moist air to rise and clouds to form that lead to precipitation, then additional routines are activated that are added to the dynamical core as parameterisations — these include cloud processes, turbulence, and radiation.
The researchers in the KILOS project are working closely with ETH Professor Torsten Hoefler and his team from the Scalable Parallel Computing Lab on linking the parameterisations with the dynamical core and the transformation from Python via GT4Py to CUDA or another hardware-specific language. The new developments are being tested at CSCS, on "Piz Daint" and its successor "Alps", as well as on the Finnish pre-exascale system LUMI, currently Europe's fastest supercomputer and one in which Switzerland holds shares that it can provide to Swiss researchers.
Laseyer winds in Appenzell serve as a test case
As a first test application of the new model, Nicolai Krieger, a doctoral student in Wernli's research group, is investigating a flow phenomenon in Appenzellerland in north-eastern Switzerland. The region, especially the narrow Schwendetal valley, is known for the dreaded Laseyer winds, which can reach gust speeds of over 150 kilometres per hour — capable of lifting train wagons off the tracks. For this reason, ETH Zurich, MeteoSwiss, and Appenzeller Bahnen together developed an early warning system that makes it possible to switch from rail traffic to bus service reliably and in good time. As part of the KILOS project, the new high-resolution FVM model is being used to precisely research the wind system, which is still not fully understood.
For the time being, the newly developed model will be used for such research projects by Wernli and his team at ETH Zurich. Additionally, the FVM will be further developed until it is as reliable as the IFS model currently used by the ECMWF. For Wernli, however, the project has already fulfilled a long-standing wish: "It is a matter of concern to me that climate and weather researchers work together more," says Wernli. The KILOS project and its team are therefore in close exchange with the EXCLAIM research initiative, launched by ETH Zurich together with CSCS, MeteoSwiss, and other partners with the aim of significantly increasing the spatial resolution of weather and climate models in order to increase their precision.
Parallel to the code development in KILOS, other models are also being optimised in EXCLAIM. In both projects, the focus is on high-resolution and highly efficient models that are also useful for climate research, as they represent elementary processes such as storms and other weather systems in much greater detail. This makes it possible to investigate how climate change affects specific weather events, and it also promises a future with more accurate, timely, hyper-local weather forecasts.
Image above: Monday, 28 August 2023 - The Rhine in Diepoldsau, Switzerland, carries a lot of water after torrential rains. (Image: MeteoSwiss App, Meteo Meldung)
The European Centre for Medium-Range Weather Forecasts (ECMWF), based in Reading, UK, has been one of the world's leading weather forecasting centres since 1975. Around 430 scientists currently work at the ECMWF with the aim of improving global weather forecasts. In order to be able to make forecasts for the next ten days, the ECMWF collects measurement data from ground weather houses, weather balloons, aircraft, ships, and satellites worldwide every six hours. These data are analysed at the ECMWF and fed into the model as a homogeneously gridded data set as the starting condition for the forecast. MeteoSwiss uses this data to produce daily weather forecasts and makes the ECMWF data available to researchers in Switzerland. These data are essential for the diverse research projects of former ECMWF Fellow Heini Wernli and his working group. The observation dataset is also relevant for climate research, because it can be used to determine whether and how certain phenomena, such as the jet stream in the atmosphere, have changed over the past 40 years.
Kühnlein C, Ehrengruber T, Ubbiali S, Krieger N, Papritz L, Calotoiu A, and Wernli H: ECMWF collaborates with Swiss partners on GPU porting of FVM dynamical core, ECMWF Newsletter, 175, 11-12, 2023.