Energy-aware operation of HPC systems in Germany

Estela Suarez ^1,2,3* Hendryk Bockelmann ⁴ Norbert Eicker ^1,5 Jan Eitzinger ⁶ Salem El Sayed ¹ Thomas Fieseler ¹ Martin Frank ⁷ Peter Frech ¹ Pay Giesselmann ⁴ Daniel Hackenberg ⁸ Georg Hager ⁶ Andreas Herten ¹ Thomas Ilsche ⁸ Bastian Koller ⁹ Erwin Laure ¹⁰ Cristina Manzano ¹ Sebastian Oeste ⁸ Michael Ott ¹¹ Klaus Reuter ¹⁰ Ralf Schneider ⁹ Kay Thust ¹ Benedikt von St. Vieth ¹

• ¹ Jülich Supercomputing Center, Institute for Advanced Simulation, Julich Research Center, Helmholtz Association of German Research Centers (HZ), Jülich, Germany
• ² SiPEARL GmbH, Duisburg, Germany
• ³ Institute of Computer Science, University of Bonn, Bonn, North Rhine-Westphalia, Germany
• ⁴ German Climate Computing Centre (MPG), Hamburg, Hamburg, Germany
• ⁵ University of Wuppertal, Wuppertal, North Rhine-Westphalia, Germany
• ⁶ High Performance Computing Center, Department of Computer Science, Friedrich Alexander University Erlangen-Nuremberg, Erlangen, Bavaria, Germany
• ⁷ Karlsruhe Institute of Technology (KIT), Karlsruhe, Baden-Württemberg, Germany
• ⁸ Technical University Dresden, Dresden, Lower Saxony, Germany
• ⁹ High Performance Computing Center Stuttgart, University of Stuttgart, Stuttgart, Baden-Württemberg, Germany
• ¹⁰ Max Planck Computing and Data Facility, Garching, Germany
• ¹¹ Leibniz Supercomputing Centre, Garching, Germany

High Performance Computing (HPC) systems are among the most energy-intensive scientific facilities, with electric power consumption reaching and often exceeding 20 Megawatts per installation. Unlike other major scientific infrastructures such as particle accelerators or highintensity light sources, which are few around the world, the number and size of supercomputers are continuously increasing. Even if every new system generation is more energy efficient than the previous one, the overall growth in size of the HPC infrastructure, driven by a rising demand for computational capacity across all scientific disciplines, and especially by Artificial Intelligence (AI) workloads, rapidly drives up the energy demand. This challenge is particularly significant for HPC centres in Germany, where high electricity costs, stringent national energy policies, and a strong commitment to environmental sustainability are key factors. This paper describes various state-of-the-art strategies and innovations employed to enhance the energy efficiency of HPC systems within the national context. Case studies from leading German HPC facilities illustrate the implementation of novel heterogeneous hardware architectures, advanced 1 Suarez et al.monitoring infrastructures, high-temperature cooling solutions, energy-aware scheduling, and dynamic power management, among other optimisations. By reviewing best practices and ongoing research, this paper aims to share valuable insight with the global HPC community, motivating the pursuit of more sustainable and energy-efficient HPC architectures and operations.