Skip to main content

PERSPECTIVE article

Front. Astron. Space Sci., 22 March 2023
Sec. Space Physics
This article is part of the Research Topic The Future of Space Physics 2022 View all 66 articles

Using Application Usability Levels to support tracking the health of Heliophysics

  • 1Goddard Space Flight Center, National Aeronautics and Space Administration, Greenbelt, MD, United States
  • 2Department of Earth, Planetary, and Space Sciences, University of California Los Angeles, Los Angeles, CA, United States

The Application Usability Level (AUL) framework describes how a project advances from basic research to operation-ready applications. Here we expand upon the current project-level usage of Application Usability Levels into a programmatic usage which can be used to help funding agencies track the health of the Heliophysics program. Examples are discussed throughout Heliophysics to show the utility of the Application Usability Level framework for describing the usability level of projects.

1 Introduction

Application Usability Levels (AULs) are used and often described within a research or research-to-operations context (e.g., Halford et al., 2019). While still a new framework, AULs have proven helpful to the community and have steadily grown in acceptance. For example, the United Kingdom Met office has now incorporated their use, and the United Kingdom and United States proposal calls for space weather applications also directly reference the AUL framework (e.g., Hapgood et al., 2020). However, this framework does not just describe progress at the project level, but can be used to support assessment of the health of the field overall.

This paper will outline how the AUL framework can be generalized to a funding organizational level. This extension was built in at the foundations of the framework, as part of its heritage from the Application Readiness Levels (ARLs). ARLs were developed by the Applied Sciences Program under the NASA HQ Earth Science Division (NASA Earth Science Applied Science, 2021). They have been successfully used in Earth Science for an extended period of time to track the health of the division and the roadblocks faced (e.g., Friedl and Sylak-Glassman, 2020). A framework such as the AULs or the ARLs can help guide strategic planning for new missions and new proposal calls. While the discussion will primarily focus on the AUL framework, these recommendations can apply to other readiness level frameworks, such as Technology Readiness Levels (e.g., Olechowski et al., 2015) and the Readiness Levels used by the National Oceanic and Atmospheric Administration (National Oceanic and Atmospheric Administration, 2021).

The AUL framework—when used across projects—provides a unique insight into the health of a program and where there may be common needs across the community. An overview of the phases is provided in Table 1. A healthy program should include projects across the AUL framework, ranging from exploratory research (AUL 1 through AUL 3) to research projects that are in their finalization and on-demand usage phases (AUL 6—AUL 9). The AUL framework provides an inventory of the balance across development levels, which can help determine whether the existing distribution of projects matches the needs of the community and funding agencies. In general, there are relatively many projects funded for early phase research, while a fraction of these reach the later phases. While it is expected that not every project will advance to an operational level, in general advancement of projects from mid to high level AULs is underfunded, resulting in the “Valley of Death” for research-to-operations (Merceret et al., 2013).

TABLE 1
www.frontiersin.org

TABLE 1. A brief description of the AUL phases and levels.

2 Current examples of AULs in Heliophysics

One advantage of the AUL concept is that is a generalized framework that can be used for a variety of project types. Below are several examples of how this framework is being used in Heliophysics across the different phases outlines in Table 1.

2.1 Proof of concept studies

Phase 1 of the AUL framework covers the discovery and viability of possible applications, including proof of concept studies. While this phase is most similar to typical research projects, there is the added step of documenting the usage beyond immediate research. An example of a current Phase 1 project discussing potential applications is a new model of Solar Energetic Particle (SEP) injection and transport. van den Berg et al. (2020) outlines a list of requirements for predictive capability of SEPs needed for future crewed missions and uses this to inform the direction of future research (AUL 2). The potential use of solar soft X-ray activity as a proxy for SEPs is proposed as a proof-of-concept study (Steyn et al., 2020), which is consistent with AUL 3 (assess viability and current state of the art).

2.2 Prediction and validation of space weather events using models

Extending into Phase 2, testing and validation is now required. This describes projects that have moved beyond basic research and into applications. For example, the routine prediction of the growth of night-time Equatorial Plasma Bubbles is a space weather problem that has been outstanding for over 80 years due to the complex interplay of physical forces that can enhance or suppress the likelihood of growth on a given night (e.g., Klenzing et al., 2023). Recent work to improve these predictions has been outlined in Section 4.4 of Halford et al. (2019) using the AUL framework. An extensive validation campaign to compare 12 months of ground-based data with long-term runs of models was recently conducted as part of the validation track of the AUL framework (Carter et al., 2020).

2.3 Development of new space weather indices

Phase 3 projects have completed validation and are being implemented into operational solutions. The AUL framework has been employed in the development and validation of the LDi and LCi geomagnetic indices (Cid et al., 2020; Nahayo et al., 2022). These new indices were developed jointly by researchers and industry users for nowcasting local geomagnetic disturbances (LDi) and for geomagnetically induced current risk (LCi). They are currently being deployed real-time on ground-based stations (Guerrero et al., 2021).

2.4 Software development

The AUL framework has motivated the discussion of new metrics for model assessment (Liemohn et al., 2020, 2021; 2022). Additionally, it is useful for software projects that may have multiple applications, and therefore different AUL progress depending on the usage. For example, the sami2py software project consists of an open source python package that runs the SAMI2 (Sami2 is Another Model of the Ionosphere) ionospheric model (Huba et al., 2000) and archives the results. For a given version of the software, the different applications can be rated at different levels. For example, the usage of the software in early phase research projects was rated at a level of AUL 7 (application prototype) as this is actively being used by researchers, while the usage of the software as a educational tool is rated as Phase 1 since the needs of a broader user base (i.e., students from a wider range of computational backgrounds) is still actively being identified (Klenzing et al., 2022). These new user requirements will inform the future versions and releases of the software.

3 How the AULs can help funding agencies

We recommend that the AULs be used in a similar manner to how the ARLs have been successfully used within the Earth Science program (see Section 4. Program Performance and Budget; Friedl and Sylak-Glassman, 2020). Several examples and recommendations are given below.

3.1 Healthy program—Projects across levels

The AULs can help show that the funding programs are healthy in the respect of funding projects and research across all levels of development. A balanced portfolio may then encapsulate research proposals which will achieve 1 or 2 increases in AUL phases (e.g., a project that starts at AUL 3 and will reach an AUL 4 within the 3 years), as well as research projects which span the AUL pipeline (AULs 1–9). Looking at the funding elements in this fashion can help identify gaps in the development process funded by the current portfolio.

For example, the NASA Heliophysics program currently has Guest Investigator (NASA, 2023b) and Supporting Research (NASA, 2023c) programs which focus on basic research through analysis products which would be used by other scientists. The Living With a Star program (NASA, 2023e) supports early phase AUL research through more targeted research moving through the first few levels. The Heliophysics Data Environment Enhancement (NASA, 2023a) and Heliophysics Tools and Methods (NASA, 2023d) programs target new data products and code usability (Phases 2 and 3). Additional elements could be added as new needs are identified.

3.2 Identification of need for new models and techniques

While models and new analysis techniques have always been critical to the operation of Space Weather forecasting, they are now being incorporated in a more fundamental way to Heliophysics missions. For example, the ICON (Ionospheric CONnection) mission (Immel et al., 2018) incorporates high level geophysical models driven by ICON’s thermospheric measurements both as part of the mission science goals and as a publicly available community data product (Huba et al., 2017; Maute, 2017). During the development process of future missions with many distributed measurements—such as the Geospace Dynamics Constellation (Jaynes et al., 2019)—the incorporation of new models, coupling between models, and assimilation techniques will become even more critical. AULs offer a common language to discuss the usability of current models and techniques in a similar fashion to how the readiness of instrumentation is currently discussed with respect to mission development.

3.3 Identification of need for validation efforts and new data/missions

The AUL framework includes clear points where validation efforts are needed to be completed prior to advancement. In the framework outlined in Halford et al. (2019), a project can only claim an AUL level if it has completed all the milestones associated with it, regardless of whether work has been done on advanced milestones. For example, a project with an operational prototype (part of AUL 7) that has not undergone validation efforts (one of the milestones in AUL 6) can only be assessed at a maximum AUL level of 5, since the milestones for AUL 6 are not complete. This provides motivation for increased validation efforts within the community as well as a feedback mechanism for funding agencies. For example, if many projects are stuck at AUL 5 due to the lack of sufficient data, tools, or computational needs, this signals the need for new missions or new proposal calls to be initiated.

Another factor to consider is whether the AUL of a project may decrease over time. For example, projects that rely on real-time data inputs may lose access to these if an instrument ceases operation. A replacement data input would then be required, and the AUL would drop until validation and verification are completed again. It is important that the AULs of projects be periodically evaluated as part of the diagnostic efforts.

4 Discussion

The authors note that the AUL framework should be treated as a living framework, adjusting to user and researcher needs. The flexibility of this framework is key to its deployability across multiple projects. However, this flexibility also requires community engagement and continued discussion. Workshop sessions around roadblocks and new requirements and applications can also be guided by bringing researchers, developers, and end users together.

Here we have described how the AUL framework is generalizable to an organizational funding level. The use of ARLs by the Earth Science division shows an example of how a tracking framework can help ensure a healthy applied science funding line. We propose that AULs can act in a similar fashion for Heliophysics, tracking projects across maturity and helping find gaps in data observations and models that can benefit from new calls or new funding. By bringing the community together regularly, the AULs can be updated, and insights into common roadblocks can be found.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

AH wrote a version of this manuscript as a Decadal Survey white paper, and JK expanded and revised the focus. All authors contributed to manuscript concept and revision, including reading and approving the submitted version.

Funding

This work has been completed as part of the Heliophysics Internal Scientist Funding Model (HISFM) Space Precipitation Impacts (SPI) group.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Carter, B. A., Currie, J. L., Dao, T., Yizengaw, E., Retterer, J., Terkildsen, M., et al. (2020). On the assessment of daily equatorial plasma bubble occurrence modeling and forecasting. Space weather. 18, e2020SW002555. doi:10.1029/2020sw002555

CrossRef Full Text | Google Scholar

Cid, C., Guerrero, A., Saiz, E., Halford, A. J., and Kellerman, A. C. (2020). Developing the LDi and LCi geomagnetic indices, an example of application of the AULs framework. Space weather. 18, e2019SW002171. doi:10.1029/2019sw002171

CrossRef Full Text | Google Scholar

Friedl, L., and Sylak-Glassman, E. (2020). 2020 annual summary: NASA Earth science applied sciences program. Available at: https://appliedsciences.nasa.gov/sites/default/files/2021-05/ESD.Applied%20Sciences%20-%202020%20Annual%20Summary%20%281%29.pdf.

Google Scholar

Guerrero, A., Cid, C., García, A., Domínguez, E., Montoya, F., and Saiz, E. (2021). The space weather station at the University of Alcala. J. Space Weather Space Clim. 11, 24. doi:10.1051/swsc/2021007

CrossRef Full Text | Google Scholar

Halford, A. J., Kellerman, A. C., Garcia-Sage, K., Klenzing, J., Carter, B. A., McGranaghan, R. M., et al. (2019). Application usability levels: A framework for tracking project product progress. J. Space Weather Space Clim. 9, A34. doi:10.1051/swsc/2019030

CrossRef Full Text | Google Scholar

Hapgood, M., Angling, M., Attrill, G., Bisi, M., Burnett, C., Cannon, P., et al. (2020). Summary of space weather worst-case environments. 2nd revised edition. Swindon, UK: Science and Technology Facilities Council. Technical Report RAL-TR-2020-005.

Google Scholar

Huba, J. D., Joyce, G., and Fedder, J. A. (2000). Sami2 is another model of the ionosphere (Sami2): A new low-latitude ionosphere model. J. Geophys. Res. Space Phys. 105, 23035–23053. doi:10.1029/2000ja000035

CrossRef Full Text | Google Scholar

Huba, J. D., Maute, A., and Crowley, G. (2017). Sami3_ICON: Model of the ionosphere/plasmasphere system. Space Sci. Rev. 212, 731–742. doi:10.1007/s11214-017-0415-z

PubMed Abstract | CrossRef Full Text | Google Scholar

Immel, T. J., England, S. L., Mende, S. B., Heelis, R. A., Englert, C. R., Edelstein, J., et al. (2018). The ionospheric connection explorer mission: Mission goals and design. Space Sci. Rev. 214, 13. doi:10.1007/s11214-017-0449-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Jaynes, A., Ridley, A., Bishop, R., Heelis, R., Zesta, E., Anderson, B., et al. (2019). NASA science and Technology definition team for the Geospace Dynamics constellation final report. Washington, DC: NASA.

Google Scholar

Klenzing, J., Halford, A., Liu, G., Smith, J. M., Zhang, Y., Zawdie, K., et al. (2023). A system science perspective of the drivers of equatorial plasma bubbles. Front. Astronomy Space Sci. 2022, 1064150. doi:10.3389/fspas.2022.1064150

CrossRef Full Text | Google Scholar

Klenzing, J., Smith, J. M., Halford, A. J., Huba, J. D., and Burrell, A. G. (2022). Sami2py — overview and applications. Front. Astronomy Space Sci. 2022, 1066480. doi:10.3389/fspas.2022.1066480

CrossRef Full Text | Google Scholar

Liemohn, M. W., Adam, J. G., and Ganushkina, N. Y. (2022). Analysis of features in a sliding threshold of observation for numeric evaluation (STONE) curve. Space weather. 20, e2022SW003102. doi:10.1029/2022sw003102

CrossRef Full Text | Google Scholar

Liemohn, M. W., Azari, A. R., Ganushkina, N. Y., and Rastätter, L. (2020). The STONE curve: A ROC-derived model performance assessment tool. Earth Space Sci. 7, e2020EA001106. doi:10.1029/2020ea001106

PubMed Abstract | CrossRef Full Text | Google Scholar

Liemohn, M. W., Shane, A. D., Azari, A. R., Petersen, A. K., Swiger, B. M., and Mukhopadhyay, A. (2021). RMSE is not enough: Guidelines to robust data-model comparisons for magnetospheric physics. J. Atmos. Solar-Terrestrial Phys. 218, 105624. doi:10.1016/j.jastp.2021.105624

CrossRef Full Text | Google Scholar

Maute, A. (2017). Thermosphere-Ionosphere-electrodynamics general circulation model for the ionospheric connection explorer: TIEGCM-ICON. Space Sci. Rev. 212, 523–551. doi:10.1007/s11214-017-0330-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Merceret, F. J., O’Brien, T., Roeder, W. P., Huddleston, L. L., Bauman, W. H., and Jedlovec, G. J. (2013). Transitioning research to operations: Transforming the “valley of death” into a “valley of opportunity”. Space weather. 11, 637–640. doi:10.1002/swe.20099

CrossRef Full Text | Google Scholar

Nahayo, E., Guerrero, A., Lotz, S., Cid, C., Tshisaphungo, M., and Saiz, E. (2022). Validating the LDi and LCi indices in the southern hemisphere. Space weather. 20, e2022SW003092. doi:10.1029/2022sw003092

CrossRef Full Text | Google Scholar

NASA Earth Science Applied Science (2021). Expanded ARL definitions. Washington, DC: NASA. Available at: https://appliedsciences.nasa.gov/sites/default/files/2021-02/ExpandedARLDefinitions4813.pdf.

Google Scholar

NASA (2023a). Heliophysics data environment enhancements, NNH23ZDA001N-hdee. Washington, DC: NASA. Available at: https://nspires.nasaprs.com/external/solicitations/summary!init.do?solId={FFAEE381-B6C0-F9A4-0CE1-66169DACD9E8}.

Google Scholar

NASA (2023b). Heliophysics guest investigators-open, NNH23ZDA001N-HGIO. Washington, DC: NASA. Available at: https://nspires.nasaprs.com/external/solicitations/summary.do?solId={F67E67A4-E3F6-D0DD-CCAD-2E57D747BAC2}.

Google Scholar

NASA (2023c). Heliophysics supporting research, NNH23ZDA001N-HSR. Washington, DC: NASA. Available at: https://nspires.nasaprs.com/external/solicitations/summary!init.do?solId={1FE03DBA-61BD-B248-316B-60ECB612CF2F}.

Google Scholar

NASA (2023d). Heliophysics tools and methods, NNH23ZDA001N-HTM. Washington, DC: NASA. Available at: https://nspires.nasaprs.com/external/solicitations/summary!init.do?solId={17553B8F-C32A-1AD2-2A61-78670D6D21F4}.

Google Scholar

NASA (2023e). Living with a star science, NNH23ZDA001N-LWS. Washington, DC: NASA. Available at: https://nspires.nasaprs.com/external/solicitations/summary!init.do?solId={EEA99B13-6209-5D69-4284-524A4B80ADBE}.

Google Scholar

National Oceanic and Atmospheric Administration (2021). NAO 216-105B: Policy on research and development transitions. Washington, DC: NOAA. Available at: https://www.noaa.gov/organization/administration/nao-216-105b-policy-on-research-and-development-transitions.

Google Scholar

Olechowski, A., Eppinger, S. D., and Joglekar, N. (2015). “Technology readiness levels at 40: A study of state-of-the-art use, challenges, and opportunities,” in 2015 Portland International Conference on Management of Engineering and Technology (PICMET) (Portland, OR, USA: IEEE), 2084–2094. doi:10.1109/PICMET.2015.7273196

CrossRef Full Text | Google Scholar

Steyn, R., Strauss, D. T., Effenberger, F., and Pacheco, D. (2020). The soft X-ray Neupert effect as a proxy for solar energetic particle injection - a proof-of-concept physics-based forecasting model. J. Space Weather Space Clim. 10, 64. doi:10.1051/swsc/2020067

CrossRef Full Text | Google Scholar

van den Berg, J., Strauss, D. T., and Effenberger, F. (2020). A primer on focused solar energetic particle transport. Space Sci. Rev. 216, 146. doi:10.1007/s11214-020-00771-x

CrossRef Full Text | Google Scholar

Keywords: space weather, Heliophysics, application usability level, operations, funding agencies

Citation: Klenzing J, Halford AJ, Kellerman A and Thompson B (2023) Using Application Usability Levels to support tracking the health of Heliophysics. Front. Astron. Space Sci. 10:1144053. doi: 10.3389/fspas.2023.1144053

Received: 13 January 2023; Accepted: 08 March 2023;
Published: 22 March 2023.

Edited by:

Nicholas Pedatella, National Center for Atmospheric Research, United States

Reviewed by:

Michael G. Henderson, Los Alamos National Laboratory (DOE), United States
Steven Hill, NOAA Space Weather Prediction Center, United States

Copyright © 2023 Klenzing, Halford, Kellerman and Thompson. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Jeff Klenzing, jeff.klenzing@nasa.gov

These authors have contributed equally to this work and share first authorship

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.