AUTHOR=Milan Petro Junior , Rong Hongqian , Michaud Craig , Layad Naoufal , Liu Zhengchun , Coffee Ryan TITLE=Enabling real-time adaptation of machine learning models at x-ray Free Electron Laser facilities with high-speed training optimized computational hardware JOURNAL=Frontiers in Physics VOLUME=10 YEAR=2022 URL=https://www.frontiersin.org/journals/physics/articles/10.3389/fphy.2022.958120 DOI=10.3389/fphy.2022.958120 ISSN=2296-424X ABSTRACT=

The emergence of novel computational hardware is enabling a new paradigm for rapid machine learning model training. For the Department of Energy’s major research facilities, this developing technology will enable a highly adaptive approach to experimental sciences. In this manuscript we present the per-epoch and end-to-end training times for an example of a streaming diagnostic that is planned for the upcoming high-repetition rate x-ray Free Electron Laser, the Linac Coherent Light Source-II. We explore the parameter space of batch size and data parallel training across multiple Graphics Processing Units and Reconfigurable Dataflow Units. We show the landscape of training times with a goal of full model retraining in under 15 min. Although a full from scratch retraining of a model may not be required in all cases, we nevertheless present an example of the application of emerging computational hardware for adapting machine learning models to changing environments in real-time, during streaming data acquisition, at the rates expected for the data fire hoses of accelerator-based user facilities.