Mitigating Disputes Originated by Multiple Discordant Systematic Reviews and Meta-Analyses: A Survey of Methodologists and Clinicians

Background: Overlapping systematic reviews (SRs) are increasingly frequent in the medical literature. They can easily generate discordant evidence, as estimates of effect sizes and their interpretation might differ from one source to another.

Objective: To analyze how methodologists and clinicians make a decision when faced with discordant evidence formalized in structured tables.

Methods: We conducted a 16-item survey exploring how methodologists and clinicians would react when presented with multiple Summary of Findings (SoF) tables (generated using the GRADE tool) derived from 4 overlapping and discordant SRs and meta-analyses on thrombolytic therapy for intermediate-risk pulmonary embolism. SoF tables reported 4 different magnitudes of effects and overall certainty. Participants were asked to provide their recommendations regarding the intervention and the reasons behind their conclusion.

Results: Of the 80 invitees, 41 (51%) participated. The majority described themselves as “somewhat familiar” or experts with SoF tables. The majority recommended the therapy (pharmacological systemic thrombolysis), grading the recommendation as weak positive. Certainty of evidence and benefit-risk balance were the two criteria that prevailed in generating the recommendation. When faced with overlapping meta-analyses, the preferred approach was to use only high-quality SRs and exclude redundant SRs. Several participants suggested integrating the SoF tables with additional information, such as a more comprehensive evaluation of the risk of bias of systematic reviews (71%), heterogeneity/inconsistency (68%) and studies included within each SR (62%).

Conclusion: When faced with multiple controversial SR results, the type and completeness of reported information in SoF tables affect experts' ability to make recommendations. Developers of the SoF table should consider collating key information from overlapping and potentially discordant reviews.

Introduction

In 1992 evidence-based medicine (EBM) was pronounced as a new approach for teaching and practicing clinical medicine (Evidence-Based Medicine Working Group, 1992). In the past 25 years, the EBM movement succeeded in promoting rigorous and explicit methods to identify, select and appraise relevant research studies. Systematic reviews (SRs), meta-analyses (MAs), clinical practice guidelines, health technology assessment and, more recently, overviews of systematic reviews (OSRs) are evidence synthesis designs that gained increasing popularity. SRs and MAs play a pivotal role, as they often inform other evidence synthesis designs. Guidelines, health technology assessment and OSRs are built systematically from the lower levels of evidence and provide substantially more useful information for guiding clinical decision-making (Alper and Haynes, 2016). For instance, many systematically derived recommendations rely on SRs that were either previously published or created de novo by guideline developers (Cook et al., 1997).

The number of SRs and MAs is increasing. For example, Hoffmann and colleagues observed a more than 20-fold increase in the number of SRs indexed on PubMed between 1990 and 2019 (Hoffmann et al., 2021). It has been suggested that the increasing number of redundant, unnecessary and misleading evidence syntheses, described by the claim “too much evidence,” is one of the reasons for a potential crisis of the EBM movement (Greenhalgh et al., 2014).

OSRs, as the next-generation overarching study design, have been proposed as a solution for the limited utility of multiple SRs (Ioannidis, 2017). OSRs are studies that allow synthesizing a large amount of evidence from multiple SRs and identifying gaps, weaknesses and biases that affect certain research fields (Puljak, 2017). However, one of the problems that OSRs can detect is the existence of discordant SRs, i.e., reviews that cover the same topic but reach different results and different conclusions (Riva et al., 2018; Augustin et al., 2020).

Summary of Findings (SoF) tables have been designed to summarize the key results of an SR or MA and evaluate the authors' confidence in the estimates of effect in evidence syntheses (Guyatt et al., 2011). However, it is unknown which information methodologists and clinicians prioritize when faced with discordant evidence from multiple SRs or MAs, and how their decision-making algorithm could be used to improve SoFs.

This study aimed to analyze how methodologists and clinicians decide when faced with discordant evidence and how the SoF table should be constructed in the event of overlapping MAs.

Methods

Ethics

The study protocol was approved by the Ethics Committee of the University of Split School of Medicine (Approval no. 2181-198-03-04-17-0064).

Study Design

This was a cross-sectional study.

Participants

This study was conducted during the year 2017 among clinicians considered opinion-leaders in the field of pulmonary embolism and methodologists skilled in EBM. We were targeting researchers with a high number of articles (i.e., more than 10) published on topics related to this survey (i.e., pulmonary embolism, systematic, reviews, SoF tables) and a high number of citations (i.e., more than 1,000), to ensure that these individuals are recognized experts in the field. Participants were recruited by using the snowballing method.

Discordant Evidence Scored by the Participants

We invited participants to complete a scenario-based survey. The participants were asked to identify the source of discordance between contrasting evidence on pharmacological systemic thrombolytic therapy for intermediate-risk pulmonary embolism and suggest how to incorporate them in developing recommendations. They were also asked to provide suggestions for an accurate and transparent reporting of legitimate controversies in the final guideline.

For this study, we prepared four SoF tables (Supplementary File 1) from our OSR about thrombolytic therapy for intermediate-risk pulmonary embolism using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) tool (Riva et al., 2018). We chose four SRs, purposively creating a sample of real SRs with different salient characteristics. Namely, we included the SR with the highest AMSTAR (A MeaSurement Tool to Assess systematic Reviews) score (Hao et al., 2015), the SR with the lowest AMSTAR score (Gao et al., 2015), the SR with the highest number of included trials (Chen and Ren, 2014) and the SR with the highest number of citations (Chatterjee et al., 2014). Four authors produced these SoF tables (LM, EP, NR, LP) in a way that each SoF was created independently by two authors, and then the results were compared and discrepancies discussed until they were resolved. Authors resolved discrepancies via discussion. Respondents did not receive information about the SRs from which the SoF tables were extracted.

Survey

We created a 16-item survey (available in Supplementary File 2) divided into three sections for this study. In the first section, there were questions about characteristics of the participants, including their sex, age, place of residence, personal description of their role and familiarity with an SoF table. In the second section, the participants were asked to make a decision based on the four presented SoFs, explaining how they reached their decision when faced with discordant evidence and how they would judge the quality of presented evidence. In the third section, there were general questions about decision-making in the event of discordant evidence from overlapping MAs and how SoF tables should be designed in the event of overlapping MAs.

Dissemination of the Survey

Prospective participants were invited by e-mail to participate in the study, which was available as an online survey at the SurveyMonkey platform (SurveyMonkey Inc., Palo Alto, CA, USA). The survey was set up as completely anonymous; it did not collect participants' internet protocol (IP) addresses. Participants received a maximum of 3 reminders before the survey was closed.

Statistics

Descriptive statistics with frequencies and percentages were used based on the built-in statistical features of SurveyMonkey. Characteristics of clinicians vs. methodologists were compared using the Chi Square or the Fisher's exact tests, as appropriate. The software STATA/BE version 17 (StataCorp LP, College Station, TX) was used for statistical analysis with two-tailed p < 0.05 considered statistically significant.

Results

Of the 80 invited participants, 41 (51%) accepted the invitation and participated in the study. A similar number of women (N = 20; 49%) and men participated. Most of the participants were aged 41 to 50. Most had a residence in Europe (29, 71%), and most of them were either methodologists (N = 22; 54%) or clinicians (N = 15; 37%). The majority of participants described themselves as ‘somewhat familiar' with the SoF tables (N = 15; 37%) or experts involved in GRADE methods and who can teach others about creating the SoF tables (N = 12; 29%) (Table 1).

TABLE 1

Table 1. Characteristics of study participants (responses provided by all 41 participants).

Based on the evidence/SoF tables presented, most participants proposed that thrombolytic therapy (i.e., infusion into a vein of a medicine that will break up or dissolve blood clots, in addition to an anticoagulant medicine) “is suggested” or “might be suggested” in patients with pulmonary embolism at intermediate-risk of death at 30 days. They also mostly agreed that the grade for this recommendation was weak positive. Most of them (N = 27; 79%) based their recommendation on two or more SoF tables. They identified the most important outcome (i.e., mortality) and then considered that the quality of evidence for this outcome was marked in the preferred SoF tables as “high.” They reported that the certainty of the evidence and the benefit-risk balance were two criteria that most often guided their choice of recommendation (Table 2).

TABLE 2

Table 2. Participants' recommendation regarding the intervention, and rationale behind their decision-making (responses provided by 34 participants).

When faced with overlapping MAs, which may reach different conclusions, the participants suggested using only high-quality SRs and excluding redundant reviews as the most effective strategy in supporting guideline development (N = 25; 76% of the participants gave this option the highest priority mark) (Figure 1).

FIGURE 1

Figure 1. Participants' opinion about approaches to be taken for supporting guideline development group decisions in case of overlapping meta-analyses (responses provided by 34 participants). Figure shows responses to the following question: Overlapping meta-analyses can often be confusing because they may reach different conclusions. In such cases, which approach could be the most effective in supporting guideline development group decisions? Please order the statements by relevance, the most relevant = 1.

In case of overlapping MAs, in addition to the SoFs, additional information that the participants would like to have were: 1. the risk of bias in each review (measured with AMSTAR or a tool for assessing the risk of bias in systematic reviews—ROBIS) (N = 24; 71%), 2. the heterogeneity / consistency among the results of included studies/SRs (N = 23; 68%), and 3. the included studies within each SR (N = 21; 62%) (Figure 2). In addition, 41% of the participants indicated that they do not think that the actual form of the SoF table supports users in capturing differences across overlapping SRs (Figure 3). However, a minority did not consider overlapping SRs a problem deserving a dedicated approach in GRADE. Some participants indicated that it is not possible to improve the SoF to present evidence from overlapping reviews, mainly because clinical decisions should be based on one SR, which is the most up-to-date, or the one with the highest quality, accompanied by a clear justification for this choice (Table 3).

FIGURE 2

Figure 2. Suggestions of participants regarding information needed for decision-making in case of overlapping systematic reviews (responses provided by 34 participants). Figure shows responses to the following question: In case of overlapping meta-analyses, on top of summary of findings table, which additional information would you like to have? (multiple answers allowed). Y-axis denotes percentage of participants.

FIGURE 3

Figure 3. The participants' opinion on whether the actual form of the summary of findings table captures differences across overlapping systematic reviews (responses provided by 34 participants).

TABLE 3

Table 3. Categorized suggestions to revise the summary of findings template in case of overlapping reviews (open-ended response).

An open-ended question about ideas to improve the SoF in case of overlapping SRs yielded 19 responses (Table 3). Participants mostly provided suggestions for improving the SoF table by having the option to present the results of the overlapping SRs. Most of them suggested that various information should be added to the SoF, which may help in discerning reasons for discrepancies.

When we compared responses between clinicians and methodologists, we found that methodologists were more often experts or very familiar with the SoF. There were no significant differences in the type of recommendation these two groups provided based on the evidence presented and their grade of recommendation (Table 4). There was no difference in the frequency of the chosen criteria related to the SoF table, which guided the decision making. Methodologists more commonly indicated that they would need detailed information on the included studies within each systematic review to make a decision (Table 4).

TABLE 4

Table 4. Differences in responses between clinicians and methodologists.

We published raw data from this study on the Open Science Framework; the raw data table is available on the following link: https://osf.io/qsbyh/.

Discussion

We presented different SoF tables from overlapping SRs to methodologists and clinicians, who were somewhat familiar or experts with GRADE methodology and the development of SoF tables. Even though they addressed the same clinical question, i.e., systemic thrombolysis for intermediate-risk pulmonary embolism, the SoF tables reported results that were contradictory to some extent. Nevertheless, based on the quality of the SRs, participants were able to identify a recommendation-making process, giving preference to high-quality reviews. When faced with discordant evidence, they were more likely to formulate a weak recommendation. Additional information relevant to participants for supporting the clinical decision-making process were risk of bias, heterogeneity, and information about included studies in discordant SRs.

In 1997, the question of interpreting discordant results from similar SRs was first addressed (Jadad et al., 1997). It has been documented that escalation in the number of SRs increases the likelihood of finding conflicting evidence (Lucenteforte et al., 2015; Riva et al., 2018). These disputes between different SRs on the same topic may lead to the dissemination of inconsistent recommendations, slowing the transfer of research evidence into practice.

OSRs can easily identify cases of overlapping and discordant reviews. However, a recent analysis by Lunny and colleagues of 50 randomly sampled overviews showed that only a minority of them reported methods for handling overlapping reviews and discordant results and/or conclusions (Lunny et al., 2020). Another challenge for information users is also to appraise whether some of the discordant SRs meet the criteria of a SRas one or more may not be systematic in part or at all (Dettori and Norvell, 2020). Several typologies of literature reviews have been proposed (Par et al., 2015; Munn et al., 2018), contributing to different definitions of SR over time and across fields. The lack of a consensual definition of an SR and agreement on core methodological elements (e.g., identification of all relevant records, selection of eligible studies, assessment of the risk of bias, qualitative synthesis of the included studies, and meta-analyses when possible) can challenge an explicit and reproducible selection of SRs by researchers (Krnic Martinic et al., 2019).

This survey offers preliminary insights on concepts and practices among reviewers and health professionals who read SRs, or use them in the context of recommendations development when facing multiple sources of evidence. Based on our findings, we suggest that the structure of SoF tables should be complemented with a dedicated module of information to present users with details about overlapping and potentially discordant SRs. The purpose of such a tool would be to inform the reporting of overlapping evidence syntheses rather than provide a method to identify or classify discordant evidence. It could, however, be also used by reviewers, guideline developers and journal editors to plan for a comprehensive appraisal of evidence syntheses in situations where it is important to consider overlapping sources and conflicting results.

Using a new type of SoF table for OSRs, as a tool that would help collate the essential information about overlapping and potentially discordant reviews is likely to support clinical decision-making and more transparent guideline recommendations. As Bobbio and colleagues wrote in 1991 in a seminal paper in the Lancet about optimal reporting of trial results, “the key question in most therapeutic controversies is not whether the proposed treatment is effective or not, but how data should be presented in order to allow physicians and patients to decide for themselves” (Bobbio et al., 1994). Overall, a broader appreciation of the need for reporting findings from multiple sources, particularly when they reach high-quality thresholds, would be a welcome evolution in evidence synthesis science.

Our study has several potential limitations. There is no consensus definition of an opinion leader, which might have hampered our selection of experts that assessed the SoF tables. We used snowball sampling for the recruitment of participants. Such sampling is frequently used as a form of non-random sampling where a high external validity is not sought as prominent feature of a study (Parker et al., 2019). The initial sample relies on personal contacts. Thus, with this sampling strategy, we cannot generalize our results. In our study, we compared the responses of clinicians and methodologists. As the numbers of participants in each group were small, any difference should be interpreted as hypothesis-generating. Our study targeted a single medical question, namely thrombolytic therapy for intermediate-risk pulmonary embolism, again limiting external validity. Familiarity with the topic could have influenced the responses of some participants.

In conclusion, when faced with multiple controversial SR results, the type and completeness of reported information in SoF tables could affect experts' ability to make recommendations. Therefore, developers of the SoF table should consider collating the critical information from overlapping and potentially discordant reviews.

Data Availability Statement

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found at: we published raw data on the Open Science Framework, and made it public. The raw data table is available on the following link: https://osf.io/qsbyh/.

Ethics Statement

The study protocol was approved by the Ethics Committee of the University of Split School of Medicine (Approval no. 2181-198-03-04-17-0064). The patients/participants provided their written informed consent to participate in this study.

Author Contributions

All authors study design, data analysis, data interpretation, writing of the manuscript, and approval of the final version of the manuscript.

Funding

This work was supported by the Italian Ministry of Health (Giovani Ricercatori GR-2011-02348048) and by Regione Lombardia (R.L. d.g.r. n. IX/4662, 9/01/2013).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors are grateful to the study participants for taking part in the survey.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frma.2022.849019/full#supplementary-material

References

Alper, B. S., and Haynes, R. B. (2016). EBHC pyramid 5.0 for accessing preappraised evidence and guidance. Evid. Based Med. 21, 123–125. doi: 10.1136/ebmed-2016-110447

PubMed Abstract | CrossRef Full Text | Google Scholar

Augustin, G., Boric, M., Barcot, O., and Puljak, L. (2020). Discordant outcomes of laparoscopic versus open appendectomy for suspected appendicitis during pregnancy in published meta-analyses: an overview of systematic reviews. Surg. Endosc. 34, 4245–4256. doi: 10.1007/s00464-020-07674-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Bobbio, M., Demichelis, B., and Giustetto, G. (1994). Completeness of reporting trial results: effect on physicians' willingness to prescribe. Lancet 343, 1209–1211. doi: 10.1016/S0140-6736(94)92407-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Chatterjee, S., Chakraborty, A., Weinberg, I., Kadakia, M., Wilensky, R. L., Sardar, P., et al. (2014). Thrombolysis for pulmonary embolism and risk of all-cause mortality, major bleeding, and intracranial hemorrhage: a meta-analysis. JAMA 311, 2414–2421. doi: 10.1001/jama.2014.5990

PubMed Abstract | CrossRef Full Text | Google Scholar

Chen, H., and Ren, C. (2014). Thrombolysis versus anticoagulation for the initial treatment of moderate pulmonary embolism: a meta-analysis of randomized controlled trials. Respir. Care 59, 1880–1887. doi: 10.4187/respcare.03197

PubMed Abstract | CrossRef Full Text | Google Scholar

Cook, D. J., Greengold, N. L., Ellrodt, A. G., and Weingarten, S. R. (1997). The relation between systematic reviews and practice guidelines. Ann. Intern. Med. 127, 210–216. doi: 10.7326/0003-4819-127-3-199708010-00006

PubMed Abstract | CrossRef Full Text | Google Scholar

Dettori, J. R., and Norvell, D. C. (2020). Discordant systematic reviews: which to believe? Global Spine J. 10, 237–239. doi: 10.1177/2192568219899945

PubMed Abstract | CrossRef Full Text | Google Scholar

Evidence-Based Medicine Working Group (1992). Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA 268, 2420–2425. doi: 10.1001/jama.268.17.2420

CrossRef Full Text | Google Scholar

Gao, G. Y., Yang, P., Liu, M., Ding, M., Liu, G. H., Tong, Y. L., et al. (2015). Thrombolysis for acute intermediate-risk pulmonary embolism: a meta-analysis. Thromb. Res. 136, 932–937. doi: 10.1016/j.thromres.2015.09.012

PubMed Abstract | CrossRef Full Text | Google Scholar

Greenhalgh, T., Howick, J., and Maskrey, N. (2014). Evidence based medicine: a movement in crisis? BMJ 348, g3725. doi: 10.1136/bmj.g3725

PubMed Abstract | CrossRef Full Text | Google Scholar

Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., et al. (2011). GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J. Clin. Epidemiol. 64, 383–394. doi: 10.1016/j.jclinepi.2010.04.026

PubMed Abstract | CrossRef Full Text | Google Scholar

Hao, Q., Dong, B. R., Yue, J., Wu, T., and Liu, G. J. (2015). Thrombolytic therapy for pulmonary embolism. Cochrane Database Syst Rev. 12, CD004437. doi: 10.1002/14651858.CD004437.pub4

PubMed Abstract | CrossRef Full Text | Google Scholar

Hoffmann, F., Allers, K., Rombey, T., Helbach, J., Hoffmann, A., Mathes, T., et al. (2021). Nearly 80 systematic reviews were published each day: observational study on trends in epidemiology and reporting over the years 2000-2019. J. Clin. Epidemiol. 138, 1–11. doi: 10.1016/j.jclinepi.2021.05.022

PubMed Abstract | CrossRef Full Text | Google Scholar

Ioannidis, J (2017). Next-generation systematic reviews: prospective meta-analysis, individual-level data, networks and umbrella reviews. Br. J. Sports Med. 51, 1456–1458. doi: 10.1136/bjsports-2017-097621

PubMed Abstract | CrossRef Full Text | Google Scholar

Jadad, A. R., Cook, D. J., and Browman, G. P. (1997). A guide to interpreting discordant systematic reviews. CMAJ 156, 1411–1416.

PubMed Abstract | Google Scholar

Krnic Martinic, M., Pieper, D., Glatt, A., and Puljak, L. (2019). Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med. Res. Methodol. 19, 203. doi: 10.1186/s12874-019-0855-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Lucenteforte, E., Moja, L., Pecoraro, V., Conti, A. A., Conti, A., Crudeli, E., et al. (2015). Discordances originated by multiple meta-analyses on interventions for myocardial infarction: a systematic review. J. Clin. Epidemiol. 68, 246–256. doi: 10.1016/j.jclinepi.2014.11.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Lunny, C., Brennan, S. E., Reid, J., McDonald, S., and McKenzie, J. E. (2020). Overviews of reviews incompletely report methods for handling overlapping, discordant, and problematic data. J. Clin. Epidemiol. 118, 69–85. doi: 10.1016/j.jclinepi.2019.09.025

PubMed Abstract | CrossRef Full Text | Google Scholar

Munn, Z., Stern, C., Aromataris, E., Lockwood, C., and Jordan, Z. (2018). What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Med. Res. Methodol. 18, 5. doi: 10.1186/s12874-017-0468-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Par,é, G, Trudel, M.-C., Jaana, M., and Kitsiou, S. (2015). Synthesizing information systems knowledge: A typology of literature reviews. Information Manage. 52, 183–199. doi: 10.1016/j.im.2014.08.008

CrossRef Full Text | Google Scholar

Parker, C., Scott, S., and Geddes, A. (2019). “Snowball sampling,” in SAGE Research Methods Foundations, eds P. Atkinson, S. Delamont, A. Cernat, J. W. Sakshaug, and R. A. Williams (London: Sage Publications). doi: 10.4135/9781526421036831710

CrossRef Full Text | Google Scholar

Puljak, L (2017). If there is only one author or only one database was searched, a study should not be called a systematic review. J. Clin. Epidemiol. 91, 4–5. doi: 10.1016/j.jclinepi.2017.08.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Riva, N., Puljak, L., Moja, L., Ageno, W., Schunemann, H., Magrini, N., et al. (2018). Multiple overlapping systematic reviews facilitate the origin of disputes: the case of thrombolytic therapy for pulmonary embolism. J. Clin. Epidemiol. 97, 1–13. doi: 10.1016/j.jclinepi.2017.11.012

PubMed Abstract | CrossRef Full Text | Google Scholar