AUTHOR=Norris Matthew B. , Grohs Jacob R. , Knight David B. 

TITLE=Investigating student approaches to scenario-based assessments of systems thinking

JOURNAL=Frontiers in Education

VOLUME=Volume 7 - 2022

YEAR=2022

URL=https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2022.1055403

DOI=10.3389/feduc.2022.1055403

ISSN=2504-284X

ABSTRACT=Colleges and universities continue to be pressed to develop graduates who can exhibit systems-thinking capabilities so that they can help address interdisciplinary complex problems. Although important for graduates of all fields, systems thinking has been identified as especially important within the engineering field. Engineers are tasked with solving socio-technical problems that often require knowledge beyond their original discipline, making critical systems thinking skills that enable them to identify relevant knowledge across traditional disciplinary boundaries. In order to provide the developmental feedback necessary to teach these skills at an individual level, instructors need to be able to quickly and accurately assess students, yet existing assessments often fail to accurately measure teachable knowledge and skills. Scenario-based assessments have been proposed as a potential avenue of assessment, requiring less time investment than simulations or design projects while still offering more detailed assessment than self-reports. However, the scope and domain of these problems can vary greatly between instruments and often produce conflicting evaluations of students’ systems thinking ability. This discrepancy necessitates a closer examination of the variation across how students are engaging in solving these problems and how they leverage prior knowledge to address them.
To investigate these questions, we compared students’ performance on two previously and independently peer-reviewed scenario-based assessments for systems thinking: The Village of Abeesee and the Lake Urmia Vignette. Twenty undergraduate engineering students participated in a multi-phase case study investigating their thought processes for completing these scenarios. Students were explicitly asked to discuss what differences they saw in their approaches in thinking across the two scenarios and past experiences that they felt prepared them to solve these problems.
Our findings suggest the way a scenario is presented to students impacts their subsequent problem-solving approach, which complicates the evaluation of students’ systems thinking. Further, the limited opportunities students identified for the development of open-ended problem-solving raises questions about whether current curriculum is structured optimally to support the development of a critical skillset. Our results can inform future work on improving systems thinking assessment and contribute toward improving engineering curricula to more intentionally supply opportunities for students to practice solving ill-structured problems.