AUTHOR=Schmidt Philipp , Ahmed Karim , Danilevski Cyril , Hammer David , Rosca Robert , Kluyver Thomas , Michelat Thomas , Sobolev Egor , Gelisio Luca , Maia Luis , Manetti Maurizio , Malka Janusz , Wrona Krzysztof , Sztuk-Dambietz Jolanta , Rovensky Vratko , Ramilli Marco , Duarte Nuno , Lomidze David , Dourki Ibrahym , Yousef Hazem , Senfftleben Björn , Meyer Olivier , Turcato Monica , Hauf Steffen , Aplin Steve TITLE=Turning European XFEL raw data into user data JOURNAL=Frontiers in Physics VOLUME=11 YEAR=2024 URL=https://www.frontiersin.org/journals/physics/articles/10.3389/fphy.2023.1321524 DOI=10.3389/fphy.2023.1321524 ISSN=2296-424X ABSTRACT=

The European X-ray Free Electron Laser is a research facility located close to Hamburg, offering X-ray pulses with ultra-high brilliance and femtosecond duration at megahertz repetition rates. The detection systems necessary to unlock the full scientific potential made possible by this machine poses considerable challenges both in terms of data volume and rate, as well as the interpretation of their recorded signal. To provide optimal data quality, expert and detector-specific knowledge not easily accessible to external facility users is essential, and its implementation must cope with the generated volumes. We therefore aim to perform these preparatory processing steps and offer users a dataset suitable for further analysis as the primary data product. This work describes the machinery and workflows providing this data to users in an automatic, configurable and reproducible manner, both online during the experiment, and offline for scientific analysis afterward on the way to publication.