Nicholas Conlon ^1* Aastha Acharya ¹ Jamison Mcginley ¹ Trevor Slack ¹ Camron A Hirst ¹ Marissa D'Alonzo ² Mitchell R Hebert ² Christopher Reale ² Eric W Frew ¹ Rebecca Russell ² Nisar Ahmed ¹

• ¹ University of Colorado Boulder, Boulder, United States
• ² Draper Laboratory, Cambridge, Massachusetts, United States

Introduction: Future concepts for airborne autonomy point towards human operators moving out of the cockpit and into supervisory roles. Urban Air Mobility, airborne package delivery, and military Intelligence, Surveillance, and Reconnaissance (ISR) are all actively exploring concepts or currently undergoing this transition. Supervisors of these systems will be faced with many challenges, including platforms operating outside of visual range and the need to decipher complex sensor or telemetry data in order to make informed and safe decisions with respect to the platforms and their mission. A central challenge to this new paradigm of non-co-located mission supervision is developing systems whose autonomy and internal decision-making processes are explainable and trustworthy. Methods: Competency self-assessments are methods that use introspection to quantify and communicate important information pertaining to autonomous system capabilities and limitations to human supervisors. We first discuss a computational framework for competency self-assessment, called Factorized Machine Self-Confidence (FaMSeC). Within this framework, we then define the Generalized Outcome Assessment (GOA) factor, which quantifies an autonomous system's ability to meet or exceed user specified mission outcomes. As a relevant example, we develop a competency-aware learning-based autonomous Uncrewed Aircraft System (UAS) and evaluate it within a multi-target ISR mission. Results: We present an analysis of the computational cost and performance of GOA-based competency reporting. Our results show that our competency self-assessment method can capture changes in the UAS's ability to achieve mission critical outcomes and discuss how this information can be easily communicated to human partners to inform decision-making. Discussion: We believe that competency self-assessment can enable AI/ML transparency and provide assurances that calibrate human operators to their autonomous teammate's ability to meet mission goals. This in turn can lead to informed decision-making, appropriate trust in autonomy, and overall improvements mission performance.