Fabian Kieser ^1* Paul Tschisgale ² Sophia Rauh ³ Xiaoyu Bai ³ Holger Maus ² Stefan Petersen ² Manfred Stede ³ Knut Neumann ² Peter Wulff ¹

• ¹ Heidelberg University of Education, Heidelberg, Germany
• ² Department of Physics Didactics, Leibniz Institute for Science and Mathematics Education, Faculty of Mathematics and Natural Sciences, University of Kiel, Kiel, Schleswig-Holstein, Germany
• ³ University of Potsdam, Potsdam, Brandenburg, Germany

Large language models have been shown to excel in many different tasks across disciplines and research sites. They provide novel opportunities to enhance educational research and instruction in different ways such as assessment. However, these methods have also been shown to have fundamental limitations. These relate, among others, to hallucinating knowledge, explainability of model decisions, and resource expenditure. As such, more conventional machine learning algorithms might be more convenient for specific research problems because they allow researchers more control over their research. Yet, the circumstances in which either conventional machine learning or large language models are preferable choices are not well understood.This study seeks to answer the question to what extent either conventional machine learning algorithms or a recently advanced large language model performs better in assessing students' concept use in a physics problem-solving task. We found that conventional machine learning algorithms in combination outperformed the large language model. Model decisions were then analyzed via closer examination of the models' classifications. We conclude that in specific contexts, conventional machine learning can supplement large language models, especially when labeled data is available.