We are pleased to introduce the collection Research Metrics and Analytics – Editor’s Pick 2021. This collection showcases spontaneous articles from 2021 identified by our Specialty Chief Editors Prof. Dietmar Wolfram and Prof. Zaida Chinchilla-Rodriquez. The work presented here highlights the broad diversity of research performed across the journal and aims to put a spotlight on the main areas of interest. Research presented here displays strong advances in theory and methodology with applications to compelling problems. This collection aims to further support Frontiers’ strong community of researchers.

University of Wisconsin–Milwaukee

Spanish National Research Council (CSIC)

Exploring Advances in Theory and Methodology with Applications to Compelling Problems

. Research related to the impact of posters, response to COVID-19 and climate change, quality assessment tools, views on reproducibility, reliability of Google Trends, and AI/ML research portfolios are all being studied: <ul> <li>Poster citations are rare, with only 1% of posters in the dataset having citations.</li> <li>Multi-hub conferences are proposed as a global solution to promote academic quality, creativity, enjoyment, and low-carbon sustainability.</li> <li>The Observational Study Quality Evaluation (OSQE) is a comprehensive quality assessment tool for observational studies.</li> <li>91% of respondents believe that detailed descriptions of research methodology can make published research more reproducible.</li> <li>Google Trends is reliable for surveys concerning RSVs of countries worldwide.</li> <li>127,000 research clusters derived from 1.4 billion citation links between 104.8 million documents provide insights into how research funding organizations fund different AI/ML topics.</li> </ul>

Views

Article Views

Article Downloads

Frontiers in Research Metrics and Analytics

Research Assessment

Scholarly Communication

Research Metrics and Analytics Editors' Pick 2021

We present a small case study on citations of conference posters using poster collections from both Figshare and Zenodo. The study takes into account the years 2016–2020 according to the dates of publication on the platforms. Citation data was taken from DataCite, Crossref and Dimensions. Primarily, we want to know to what extent scientific posters are being cited and thereby which impact posters potentially have on the scholarly landscape and especially on academic publications. Our data-driven analysis reveals that posters are rarely cited. Citations could only be found for 1% of the posters in our dataset. A limitation in this study however is that the impact of academic posters was not measured empirical but rather descriptive.

We present a small case study on citations of conference posters using poster collections from both Figshare and Zenodo. The study takes into account the years 2016&#x2013;2020 according to the dates of publication on the platforms. Citation data was taken from DataCite, Crossref and Dimensions. Primarily, we want to know to what extent scientific posters are being cited and thereby which impact posters potentially have on the scholarly landscape and especially on academic publications. Our data-driven analysis reveals that posters are rarely cited. Citations could only be found for 1% of the posters in our dataset. A limitation in this study however is that the impact of academic posters was not measured empirical but rather descriptive.

Are Conference Posters Being Cited?

New conference formats are emerging in response to COVID-19 and climate change. Virtual conferences are sustainable and inclusive regardless of participant mobility (financial means, caring commitments, disability), but lack face-to-face contact. Hybrid conferences (physical meetings with additional virtual presentations) tend to discriminate against non-fliers and encourage unsustainable flying. Multi-hub conferences mix real and virtual interactions during talks and social breaks and are distributed across nominally equal hubs. We propose a global multi-hub solution in which all hubs interact daily in real time with all other hubs in parallel sessions by internet videoconferencing. Conference sessions are confined to three equally-spaced 4-h UTC timeslots. Local programs comprise morning and afternoon/evening sessions (recordings from night sessions can be watched later). Three reference hubs are located exactly 8 h apart; additional hubs are within 2 h and their programs are aligned with the closest reference hub. The conference experience at each hub depends on the number of local participants and the time difference to the nearest reference. Participants are motivated to travel to the nearest hub. Mobility-based discrimination is minimized. Lower costs facilitate diversity, equity, and inclusion. Academic quality, creativity, enjoyment, and low-carbon sustainability are simultaneously promoted.

New conference formats are emerging in response to COVID-19 and climate change. Virtual conferences are sustainable and inclusive regardless of participant mobility (financial means, caring commitments, disability), but lack face-to-face contact. Hybrid conferences (physical meetings with additional virtual presentations) tend to discriminate against non-fliers and encourage unsustainable flying. Multi-hub conferences mix real and virtual interactions during talks and social breaks and are distributed across nominally equal hubs. We propose a global multi-hub solution in which all hubs interact daily in real time with all other hubs in parallel sessions by internet videoconferencing. Conference sessions are confined to three equally-spaced 4-h UTC timeslots. Local programs comprise morning and afternoon/evening sessions (recordings from night sessions can be watched later). Three reference hubs are located exactly 8&nbsp;h apart; additional hubs are within 2&nbsp;h and their programs are aligned with the closest reference hub. The conference experience at each hub depends on the number of local participants and the time difference to the nearest reference. Participants are motivated to travel to the nearest hub. Mobility-based discrimination is minimized. Lower costs facilitate diversity, equity, and inclusion. Academic quality, creativity, enjoyment, and low-carbon sustainability are simultaneously promoted.

The Multi-hub Academic Conference: Global, Inclusive, Culturally Diverse, Creative, Sustainable

Background: Existing study quality and risk of bias lists for observational studies have important disadvantages. For this reason, a comprehensive widely applicable quality assessment tool for observational studies was developed.Methods: Criteria from three quality lists were merged into a new quality assessment tool: the observational study quality evaluation (OSQE). OSQE consists of a cohort, case–control, and cross-sectional version.Results: The OSQE cohort, the OSQE case–control, and the OSQE cross-sectional version include all items applicable to that type of study, for example, the representativeness of the study population, the validity of the independent and dependent variables, and the statistical methods used. Before scoring the OSQE, the rater is asked to define how to score items, in detail. A study can obtain a star for each item. Each item also has a veto cell. This cell can be checked when poor quality with respect to that specific item results in a low quality of the study despite stars on other items. Although stars add to a sum score, the comment field is the most important part of the OSQE.Conclusion: The OSQE presented in the current article provides a short, comprehensive, and widely applicable list to assess study quality and therewith risk of bias.

Background: Existing study quality and risk of bias lists for observational studies have important disadvantages. For this reason, a comprehensive widely applicable quality assessment tool for observational studies was developed.Methods: Criteria from three quality lists were merged into a new quality assessment tool: the observational study quality evaluation (OSQE). OSQE consists of a cohort, case–control, and cross-sectional version.Results: The OSQE cohort, the OSQE case–control, and the OSQE cross-sectional version include all items applicable to that type of study, for example, the representativeness of the study population, the validity of the independent and dependent variables, and the statistical methods used. Before scoring the OSQE, the rater is asked to define how to score items, in detail. A study can obtain a star for each item. Each item also has a veto cell. This cell can be checked when poor quality with respect to that specific item results in a low quality of the study despite stars on other items. Although stars add to a sum score, the comment field is the most important part of the OSQE.Conclusion: The OSQE presented in the current article provides a short, comprehensive, and widely applicable list to assess study quality and therewith risk of bias.

Development of a Methodological Quality Criteria List for Observational Studies: The Observational Study Quality Evaluation

We constructed a survey to understand how authors and scientists view the issues around reproducibility, focusing on interactive elements such as interactive figures embedded within online publications, as a solution for enabling the reproducibility of experiments. We report the views of 251 researchers, comprising authors who have published in eLIFE Sciences, and those who work at the Norwich Biosciences Institutes (NBI). The survey also outlines to what extent researchers are occupied with reproducing experiments themselves. Currently, there is an increasing range of tools that attempt to address the production of reproducible research by making code, data, and analyses available to the community for reuse. We wanted to collect information about attitudes around the consumer end of the spectrum, where life scientists interact with research outputs to interpret scientific results. Static plots and figures within articles are a central part of this interpretation, and therefore we asked respondents to consider various features for an interactive figure within a research article that would allow them to better understand and reproduce a published analysis. The majority (91%) of respondents reported that when authors describe their research methodology (methods and analyses) in detail, published research can become more reproducible. The respondents believe that having interactive figures in published papers is a beneficial element to themselves, the papers they read as well as to their readers. Whilst interactive figures are one potential solution for consuming the results of research more effectively to enable reproducibility, we also review the equally pressing technical and cultural demands on researchers that need to be addressed to achieve greater success in reproducibility in the life sciences.

We constructed a survey to understand how authors and scientists view the issues around reproducibility, focusing on interactive elements such as interactive figures embedded within online publications, as a solution for enabling the reproducibility of experiments. We report the views of 251 researchers, comprising authors who have published in eLIFE Sciences, and those who work at the Norwich Biosciences Institutes (NBI). The survey also outlines to what extent researchers are occupied with reproducing experiments themselves. Currently, there is an increasing range of tools that attempt to address the production of reproducible research by making code, data, and analyses available to the community for reuse. We wanted to collect information about attitudes around the consumer end of the spectrum, where life scientists interact with research outputs to interpret scientific results. Static plots and figures within articles are a central part of this interpretation, and therefore we asked respondents to consider various features for an interactive figure within a research article that would allow them to better understand and reproduce a published analysis. The majority (91%) of respondents reported that when authors describe their research methodology (methods and analyses) in detail, published research can become more reproducible. The respondents believe that having interactive figures in published papers is a beneficial element to themselves, the papers they read as well as to their readers. Whilst interactive figures are one potential solution for consuming the results of research more effectively to enable reproducibility, we also review the equally pressing technical and cultural demands on researchers that need to be addressed to achieve greater success in reproducibility in the life sciences.

Knowledge and Attitudes Among Life Scientists Toward Reproducibility Within Journal Articles: A Research Survey

Background: Alongside the COVID-19 pandemic, government authorities around the world have had to face a growing infodemic capable of causing serious damages to public health and economy. In this context, the use of infoveillance tools has become a primary necessity.Objective: The aim of this study is to test the reliability of a widely used infoveillance tool which is Google Trends. In particular, the paper focuses on the analysis of relative search volumes (RSVs) quantifying their dependence on the day they are collected.Methods: RSVs of the query coronavirus + covid during February 1—December 4, 2020 (period 1), and February 20—May 18, 2020 (period 2), were collected daily by Google Trends from December 8 to 27, 2020. The survey covered Italian regions and cities, and countries and cities worldwide. The search category was set to all categories. Each dataset was analyzed to observe any dependencies of RSVs from the day they were gathered. To do this, by calling <mml:math id="m1" xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>i</mml:mi></mml:math></inline-formula> the country, region, or city under investigation and j the day its RSV was collected, a Gaussian distribution <mml:math id="m2" xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:msub><mml:mi>X</mml:mi><mml:mi>i</mml:mi></mml:msub><mml:mo>=</mml:mo><mml:mi>X</mml:mi><mml:mrow><mml:mo>(</mml:mo><mml:mrow><mml:msub><mml:mi>σ</mml:mi><mml:mi>i</mml:mi></mml:msub><mml:mo>,</mml:mo><mml:msub><mml:mrow><mml:mover accent="true"><mml:mi>x</mml:mi><mml:mo>¯</mml:mo></mml:mover></mml:mrow><mml:mi>i</mml:mi></mml:msub></mml:mrow><mml:mo>)</mml:mo></mml:mrow></mml:mrow></mml:math></inline-formula> was used to represent the trend of daily variations of <mml:math id="m3" xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:msub><mml:mi>x</mml:mi><mml:mrow><mml:mi>i</mml:mi><mml:mi>j</mml:mi></mml:mrow></mml:msub><mml:mo>=</mml:mo><mml:mi>R</mml:mi><mml:mi>S</mml:mi><mml:mi>V</mml:mi><mml:msub><mml:mi>s</mml:mi><mml:mrow><mml:mi>i</mml:mi><mml:mi>j</mml:mi></mml:mrow></mml:msub></mml:mrow></mml:math></inline-formula>. When a missing value was revealed (anomaly), the affected country, region or city was excluded from the analysis. When the anomalies exceeded 20% of the sample size, the whole sample was excluded from the statistical analysis. Pearson and Spearman correlations between RSVs and the number of COVID-19 cases were calculated day by day thus to highlight any variations related to the day RSVs were collected. Welch’s t-test was used to assess the statistical significance of the differences between the average RSVs of the various countries, regions, or cities of a given dataset. Two RSVs were considered statistical confident when <mml:math id="m4" xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mi>t</mml:mi><mml:mo><</mml:mo><mml:mn>1.5</mml:mn></mml:mrow></mml:math></inline-formula>. A dataset was deemed unreliable if the confident data exceeded 20% (confidence threshold). The percentage increase <mml:math id="m5" xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>Δ</mml:mi></mml:math></inline-formula> was used to quantify the difference between two values.Results: Google Trends has been subject to an acceptable quantity of anomalies only as regards the RSVs of Italian regions (0% in both periods 1 and 2) and countries worldwide (9.7% during period 1 and 10.9% during period 2). However, the correlations between RSVs and COVID-19 cases underwent significant variations even in these two datasets (<mml:math id="m6" xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mi>M</mml:mi><mml:mi>a</mml:mi><mml:mi>x</mml:mi><mml:mo> </mml:mo><mml:mo>|</mml:mo><mml:mi>Δ</mml:mi><mml:mo>|</mml:mo><mml:mo> </mml:mo><mml:mo>=</mml:mo><mml:mo> </mml:mo><mml:mo>+</mml:mo><mml:mo> </mml:mo><mml:mn>625</mml:mn><mml:mo>%</mml:mo></mml:mrow></mml:math></inline-formula> for Italian regions, and <mml:math id="m7" xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mi>M</mml:mi><mml:mi>a</mml:mi><mml:mi>x</mml:mi><mml:mo> </mml:mo><mml:mrow><mml:mo>|</mml:mo><mml:mi>Δ</mml:mi><mml:mo>|</mml:mo></mml:mrow><mml:mo>=</mml:mo><mml:mo> </mml:mo><mml:mo>+</mml:mo><mml:mn>175</mml:mn><mml:mo>%</mml:mo><mml:mo> </mml:mo></mml:mrow></mml:math></inline-formula> for countries worldwide). Furthermore, only RSVs of countries worldwide did not exceed confidence threshold. Finally, the large amount of anomalies registered in Italian and international cities’ RSVs made these datasets unusable for any kind of statistical inference.Conclusion: In the considered timespans, Google Trends has proved to be reliable only for surveys concerning RSVs of countries worldwide. Since RSVs values showed a high dependence on the day they were gathered, it is essential for future research that the authors collect queries’ data for several consecutive days and work with their RSVs averages instead of daily RSVs, trying to minimize the standard errors until an established confidence threshold is respected. Further research is needed to evaluate the effectiveness of this method.

Background: Alongside the COVID-19 pandemic, government authorities around the world have had to face a growing infodemic capable of causing serious damages to public health and economy. In this context, the use of infoveillance tools has become a primary necessity.Objective: The aim of this study is to test the reliability of a widely used infoveillance tool which is Google Trends. In particular, the paper focuses on the analysis of relative search volumes (RSVs) quantifying their dependence on the day they are collected.Methods: RSVs of the query coronavirus + covid during February 1—December 4, 2020 (period 1), and February 20—May 18, 2020 (period 2), were collected daily by Google Trends from December 8 to 27, 2020. The survey covered Italian regions and cities, and countries and cities worldwide. The search category was set to all categories. Each dataset was analyzed to observe any dependencies of RSVs from the day they were gathered. To do this, by calling <math id="m1"><mi>i</mi></math> the country, region, or city under investigation and j the day its RSV was collected, a Gaussian distribution <math id="m2"><mrow><msub><mi>X</mi><mi>i</mi></msub><mo>=</mo><mi>X</mi><mrow><mo>(</mo><mrow><msub><mi>σ</mi><mi>i</mi></msub><mo>,</mo><msub><mrow><mover accent="true"><mi>x</mi><mo>¯</mo></mover></mrow><mi>i</mi></msub></mrow><mo>)</mo></mrow></mrow></math> was used to represent the trend of daily variations of <math id="m3"><mrow><msub><mi>x</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>=</mo><mi>R</mi><mi>S</mi><mi>V</mi><msub><mi>s</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow></math>. When a missing value was revealed (anomaly), the affected country, region or city was excluded from the analysis. When the anomalies exceeded 20% of the sample size, the whole sample was excluded from the statistical analysis. Pearson and Spearman correlations between RSVs and the number of COVID-19 cases were calculated day by day thus to highlight any variations related to the day RSVs were collected. Welch’s t-test was used to assess the statistical significance of the differences between the average RSVs of the various countries, regions, or cities of a given dataset. Two RSVs were considered statistical confident when <math id="m4"><mrow><mi>t</mi><mo>&lt;</mo><mn>1.5</mn></mrow></math>. A dataset was deemed unreliable if the confident data exceeded 20% (confidence threshold). The percentage increase <math id="m5"><mi>Δ</mi></math> was used to quantify the difference between two values.Results: Google Trends has been subject to an acceptable quantity of anomalies only as regards the RSVs of Italian regions (0% in both periods 1 and 2) and countries worldwide (9.7% during period 1 and 10.9% during period 2). However, the correlations between RSVs and COVID-19 cases underwent significant variations even in these two datasets (<math id="m6"><mrow><mi>M</mi><mi>a</mi><mi>x</mi><mo>&nbsp;</mo><mo>|</mo><mi>Δ</mi><mo>|</mo><mo>&nbsp;</mo><mo>=</mo><mo>&nbsp;</mo><mo>+</mo><mo>&nbsp;</mo><mn>625</mn><mo>%</mo></mrow></math> for Italian regions, and <math id="m7"><mrow><mi>M</mi><mi>a</mi><mi>x</mi><mo>&nbsp;</mo><mrow><mo>|</mo><mi>Δ</mi><mo>|</mo></mrow><mo>=</mo><mo>&nbsp;</mo><mo>+</mo><mn>175</mn><mo>%</mo><mo>&nbsp;</mo></mrow></math> for countries worldwide). Furthermore, only RSVs of countries worldwide did not exceed confidence threshold. Finally, the large amount of anomalies registered in Italian and international cities’ RSVs made these datasets unusable for any kind of statistical inference.Conclusion: In the considered timespans, Google Trends has proved to be reliable only for surveys concerning RSVs of countries worldwide. Since RSVs values showed a high dependence on the day they were gathered, it is essential for future research that the authors collect queries’ data for several consecutive days and work with their RSVs averages instead of daily RSVs, trying to minimize the standard errors until an established confidence threshold is respected. Further research is needed to evaluate the effectiveness of this method.

Reliability of Google Trends: Analysis of the Limits and Potential of Web Infoveillance During COVID-19 Pandemic and for Future Research

Our work analyzes the artificial intelligence and machine learning (AI/ML) research portfolios of six large research funding organizations from the United States [National Institutes of Health (NIH) and National Science Foundation (NSF)]; Europe [European Commission (EC) and European Research Council (ERC)]; China [National Natural Science Foundation of China (NNSFC)]; and Japan [Japan Society for the Promotion of Science (JSPS)]. The data for this analysis is based on 127,000 research clusters (RCs) that are derived from 1.4 billion citation links between 104.8 million documents from four databases (Dimensions, Microsoft Academic Graph, Web of Science, and the Chinese National Knowledge Infrastructure). Of these RCs, 600 large clusters are associated with AI/ML topics, and 161 of these AI/ML RCs are expected to experience extreme growth between May 2020 and May 2023. Funding acknowledgments (in the corpus of the 104.9 million documents) are used to characterize the overall AI/ML research portfolios of each organization. NNSFC is the largest funder of AI/ML research and disproportionately funds computer vision. The EC, RC, and JSPS focus more efforts on natural language processing and robotics. The NSF and ERC are more focused on fundamental advancement of AI/ML rather than on applications. They are more likely to participate in the RCs that are expected to have extreme growth. NIH funds the largest relative share of general AI/ML research papers (meaning in areas other than computer vision, natural language processing, and robotics). We briefly describe how insights such as these could be applied to portfolio management decision-making.

Our work analyzes the artificial intelligence and machine learning (AI/ML) research portfolios of six large research funding organizations from the United States [National Institutes of Health (NIH) and National Science Foundation (NSF)]; Europe [European Commission (EC) and European Research Council (ERC)]; China [National Natural Science Foundation of China (NNSFC)]; and Japan [Japan Society for the Promotion of Science (JSPS)]. The data for this analysis is based on 127,000 research clusters (RCs) that are derived from 1.4 billion citation links between 104.8 million documents from four databases (Dimensions, Microsoft Academic Graph, Web of Science, and the Chinese National Knowledge Infrastructure). Of these RCs, 600 large clusters are associated with AI/ML topics, and 161 of these AI/ML RCs are expected to experience extreme growth between May 2020 and May 2023. Funding acknowledgments (in the corpus of the 104.9 million documents) are used to characterize the overall AI/ML research portfolios of each organization. NNSFC is the largest funder of AI/ML research and disproportionately funds computer vision. The EC, RC, and JSPS focus more efforts on natural language processing and robotics. The NSF and ERC are more focused on fundamental advancement of AI/ML rather than on applications. They are more likely to participate in the RCs that are expected to have extreme growth. NIH funds the largest relative share of general AI/ML research papers (meaning in areas other than computer vision, natural language processing, and robotics). We briefly describe how insights such as these could be applied to portfolio management decision-making.