AUTHOR=van Haastrecht Max , Sarhan Injy , Yigit Ozkan Bilge , Brinkhuis Matthieu , Spruit Marco TITLE=SYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing JOURNAL=Frontiers in Research Metrics and Analytics VOLUME=6 YEAR=2021 URL=https://www.frontiersin.org/journals/research-metrics-and-analytics/articles/10.3389/frma.2021.685591 DOI=10.3389/frma.2021.685591 ISSN=2504-0537 ABSTRACT=

Research output has grown significantly in recent years, often making it difficult to see the forest for the trees. Systematic reviews are the natural scientific tool to provide clarity in these situations. However, they are protracted processes that require expertise to execute. These are problematic characteristics in a constantly changing environment. To solve these challenges, we introduce an innovative systematic review methodology: SYMBALS. SYMBALS blends the traditional method of backward snowballing with the machine learning method of active learning. We applied our methodology in a case study, demonstrating its ability to swiftly yield broad research coverage. We proved the validity of our method using a replication study, where SYMBALS was shown to accelerate title and abstract screening by a factor of 6. Additionally, four benchmarking experiments demonstrated the ability of our methodology to outperform the state-of-the-art systematic review methodology FAST2.