The final, formatted version of the article will be published soon.
ORIGINAL RESEARCH article
Front. High Perform. Comput.
Sec. Architecture and Systems
Volume 2 - 2024 |
doi: 10.3389/fhpcp.2024.1394615
This article is part of the Research Topic Scientific Workflows at Extreme Scales View all 5 articles
ExaWorks Software Development Kit: A Robust and Scalable Collection of Interoperable Workflows Technologies
Provisionally accepted
Matteo Turilli
1,2*
Mihael Hategan-Marandiuc
3,4
Mikhail Titov
1
Ketan Maheshwari
5
Aymen Alsaadi
2
Andre Merzky
6
Justin M. Wozniak
4
Andreas Wilke
3,4
Kyle Chard
3,4
Rafael Ferreira da Silva
5
Shantenu Jha
1,2
Daniel Laney
7- 1 Brookhaven National Laboratory (DOE), Upton, New York, United States
- 2 Rutgers, The State University of New Jersey, New Brunswick, New Jersey, United States
- 3 The University of Chicago, Chicago, Illinois, United States
- 4 Argonne National Laboratory (DOE), Lemont, Illinois, United States
- 5 Oak Ridge National Laboratory (DOE), Oak Ridge, Tennessee, United States
- 6 Independent researcher, New Jersey, United States
- 7 Lawrence Livermore National Laboratory (DOE), Livermore, California, United States
Scientific discovery increasingly requires executing heterogeneous scientific workflows on highperformance computing (HPC) platforms. Heterogeneous workflows contain different types of tasks (e.g., simulation, analysis, and learning) that need to be mapped, scheduled, and launched on different computing. That requires a software stack that enables users to code their workflows and automate resource management and workflow execution. Currently, there are many workflow technologies with diverse levels of robustness and capabilities, and users face difficult choices of software that can effectively and efficiently support their use cases on HPC machines, especially when considering the latest exascale platforms. We contributed to addressing this issue by developing the ExaWorks Software Development Kit (SDK). The SDK is a curated collection of workflow technologies engineered following current best practices and specifically designed to work on HPC platforms. We present our experience with (1) curating those technologies, (2) integrating them to provide users with new capabilities, (3) developing a continuous integration platform to test the SDK on DOE HPC platforms, (4) designing a dashboard to publish the results of those tests, and (5) devising an innovative documentation platform to help users to use those technologies. Our experience details the requirements and the best practices needed to curate workflow technologies, and it also serves as a blueprint for the capabilities and services that DOE will have to offer to support a variety of scientific heterogeneous workflows on the newly available exascale HPC platforms.
Keywords: scientific workflow, Software Development Kit, high-performance computing, exascale, testing, Documentation
Received: 01 Mar 2024; Accepted: 29 May 2024.
Copyright: © 2024 Turilli, Hategan-Marandiuc, Titov, Maheshwari, Alsaadi, Merzky, Wozniak, Wilke, Chard, Ferreira da Silva, Jha and Laney. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
* Correspondence:
Matteo Turilli, Brookhaven National Laboratory (DOE), Upton, NY 11973-5000, New York, United States
Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.