Skip to main content

ORIGINAL RESEARCH article

Front. Artif. Intell.
Sec. Machine Learning and Artificial Intelligence
Volume 7 - 2024 | doi: 10.3389/frai.2024.1491932

Active learning with human heuristics: An algorithm robust to labeling bias

Provisionally accepted
Sriram Ravichandran Sriram Ravichandran 1*Nandan Sudarsanam Nandan Sudarsanam 1Balaraman Ravindran Balaraman Ravindran 1Konstantinos V. Katsikopoulos Konstantinos V. Katsikopoulos 2,3
  • 1 Indian Institute of Technology Madras, Chennai, India
  • 2 Southampton Business School, Faculty of Social Sciences, University of Southampton, Southampton, Hampshire, United Kingdom
  • 3 Department of Decision Analytics and Risk, Southampton Business School, Faculty of Social Sciences, University of Southampton, Southampton, Hampshire, United Kingdom

The final, formatted version of the article will be published soon.

    Active learning enables prediction models to achieve better performance faster by adaptively querying an oracle for the labels of data points. Sometimes the oracle is a human, for example when a medical diagnosis is provided by a doctor. According to the behavioural sciences, people, because they employ heuristics, might sometimes exhibit biases in labeling. How does modeling the oracle as a human heuristic affect the performance of active learning algorithms? If there is a drop in performance, can one design active learning algorithms robust to labeling bias? The present article provides answers. We investigate two established human heuristics (fast-and-frugal tree, tallying model) combined with four active learning algorithms (entropy sampling, multi-view learning, conventional information density, and, our proposal, inverse information density) and three standard classifiers (logistic regression, random forests, support vector machines), and apply their combinations to 15 datasets where people routinely provide labels, such as health and other domains like marketing and transportation. There are two main results. First, we show that if a heuristic provides labels, the performance of active learning algorithms significantly drops, sometimes below random. Hence, it is key to design active learning algorithms that are robust to labeling bias. Our second contribution is to provide such a robust algorithm. The proposed inverse information density algorithm, which is inspired by human psychology, achieves an overall improvement of 87% over the best of the other algorithms. In conclusion, designing and benchmarking active learning algorithms can benefit from incorporating the modeling of human heuristics.

    Keywords: Active Learning, Human in the loop, HUMAN BEHAVIOUR, biases; robustness, fast-and frugal heuristics

    Received: 05 Sep 2024; Accepted: 31 Oct 2024.

    Copyright: © 2024 Ravichandran, Sudarsanam, Ravindran and Katsikopoulos. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

    * Correspondence: Sriram Ravichandran, Indian Institute of Technology Madras, Chennai, India

    Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.