Interacting with fallible AI: Is distrust helpful when receiving AI misclassifications?

Tobias Martin Peters^*Ingrid Scharlau

• University of Paderborn, Paderborn, Germany

Due to the application of Artificial Intelligence (AI) in high-risk domains like law or medicine,trustworthy AI and trust in AI are of increasing scientific and public relevance. A typical conception,for example in the context of medical diagnosis, is that a knowledgeable user receives AI-generated classification as advice. Research to improve such interactions often aims to foster theuser’s trust, which in turn should improve the combined human-AI performance. Given that AImodels can err, we argue that the possibility to critically review, thus to distrust, an AI decision isan equally interesting target of research.We created two image classification scenarios in which the participants received mock-upAI advice. The quality of the advice decreases for a phase of the experiment. We studied thetask performance, trust and distrust of the participants, and tested whether an instruction toremain skeptical and review each piece of advice led to a better performance compared to aneutral condition. Our results indicate that this instruction does not improve but rather worsensthe participants’ performance. Repeated single-item self-report of trust and distrust shows anincrease in trust and a decrease in distrust after the drop in the AI’s classification quality, with nodifference between the two instructions. Furthermore, via a Bayesian Signal Detection Theoryanalysis, we provide a procedure to assess appropriate reliance in detail, by quantifying whetherthe problems of under- and over-reliance have been mitigated. We discuss implications of ourresults for the usage of disclaimers before interacting with AI, as prominently used in currentLLM-based chatbots, and for trust and distrust research.