Poor Quality in Systematic Reviews on PTSD and EMDR – An Examination of Search Methodology and Reporting

Background: Different user groups regard systematic reviews as reliable and valuable sources for answering research questions. For systematic reviews to fulfill their purpose, methodological quality in all stages are of importance. The studies identified in a systematic search form the basis of the review, thus the search process methodology is important for both performing and reporting the search. The purpose of the present study was to evaluate the quality of non-Cochrane systematic reviews by analyzing how they perform and report the search. This is exemplified by systematic reviews on eye movement desensitization and reprocessing (EMDR), a trauma-focused therapy commonly used for post-traumatic stress disorder (PTSD).

Methods and Results: We examined the method chapters of 20 systematic reviews on the subject, and rated their searches and reporting using relevant elements from the Cochrane Handbook and PRISMA. We found inadequacies in the methods employed for searching and reporting the search strategy, which could have been avoided by greater adherence to guiding documents for performing systematic reviews.

Conclusion: Our findings raise important questions for future debate on the risk of omitting studies, thus impairing the conclusions in a systematic review. For clinical purposes, researchers should investigate if, and how, the search strategy in a systematic review affects the body of knowledge and the results.

Introduction

Moher et al. (2007) estimated a yearly production of 2,500 systematic reviews, equaling a daily production of 6.8. By 2014 the daily production had increased to 22 (Page et al., 2016), leading to the assumption that the annual publication of systematic reviews is greater than that of randomized controlled trials (Ioannidis, 2016). The need for systematic reviews is not disputed, as they define the state of knowledge and provide a synthesis of previous research on a given issue (Cooper, 1982; Foster and Jewell, 2017), are valuable for researchers, practitioners, consumers, and policy makers (PLoS Medicine Editors, 2007; Liberati et al., 2009), inform future research (Liberati et al., 2009; Ioannidis, 2016), and help practitioners remain up-to-date in a time efficient way (Moher et al., 2007; Schlosser et al., 2007; Liberati et al., 2009; Wormald and Evans, 2018).

The Cochrane handbook provides the following explanation of what constitutes a systematic review: “A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made” (Higgins and Green, 2011). This definition distinguishes systematic reviews from other reviews, where the methodology is not explicitly stated, and the reader cannot replicate the process (Foster and Jewell, 2017).

Many systematic reviews include meta-analyses to provide estimates of effect, using statistical methods to combine data from all relevant studies (Higgins and Green, 2011). Thus, meta-analyses share recommendations for rigor, transparency, and reporting of methods with systematic reviews (Morton et al., 2018).

The time and effort required to conduct a high-quality systematic review are considerable, so it is essential to justify the need for a systematic review or meta-analysis (Wormald and Evans, 2018), avoid redundancy (Ioannidis, 2016; Cooper et al., 2018), and ensure that it is trustworthy. Authors planning to conduct systematic reviews have good sources to guide them in methodological questions, such as The Cochrane handbook (Higgins and Green, 2011), Standards for systematic reviews from the Institute of Medicine (Institute of medicine, 2011), and Methods Guide for Effectiveness and Comparative Effectiveness Reviews from the Agency for healthcare research and quality (Agency for healthcare research and quality, 2015). There are also guiding documents for critical appraisal and reporting questions for systematic reviews such as: A MeaSurement Tool to Assess systematic Reviews (AMSTAR 2) (Shea et al., 2017), and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) (Liberati et al., 2009). In addition, those new to critical appraisal can learn from systematic review checklists, such as the Critical Appraisal Skills Programme (CASP) checklist (Critical Appraisal, and Skills Programme., 2018).

The vast increase in the publication of systematic reviews can partially be explained by the following factors: for clinicians, systematic reviews can guide choices between different interventions (Page et al., 2016); systematic reviews are considered to be reliable and valuable (Bastian et al., 2010; Page et al., 2016); the authors do not need to apply for approval from an ethics committee (Wormald and Evans, 2018); and several institutions place a great deal of emphasis on the number of publications, which may drive authors to conduct and publish systematic reviews that might lack the necessary systematic rigor (Page et al., 2016). We also recognize that financing can be an issue in research projects, one of many barriers that do not apply to systematic reviews (Wormald and Evans, 2018).

The studies identified in a systematic search form the basis of the review, and the search process is of importance for the methodology for both performing and reporting the search (Liberati et al., 2009; Sampson et al., 2009; McGowan et al., 2016; Wormald and Evans, 2018). The systematic search process consists of deciding what sources to search, planning the search process, designing search strategies, managing references, and documenting and reporting the search process (Higgins and Green, 2011). Cooper et al. (2018) have identified eight key stages of the search process from central guidance documents for systematic reviews, as shown in Table 1.

TABLE 1

Table 1. Key stages of the literature search process.

This process aims to build a sound base for analyzing and synthesizing evidence, thus minimizing the risk of not including all relevant studies (Cooper et al., 2018).

Controlled vocabulary, or subject terms, is considered central for retrieving relevant records (Higgins and Green, 2011). The use of relevant subject terms in a search strategy is one of six elements that highly impact the precision of the search and the number of relevant records retrieved (Sampson et al., 2009), other elements being translation of the research question, logical operator errors, line number errors and spelling errors. Subject specific databases use controlled vocabulary to index and describe the content of an article. Medical Subject Headings (MeSH^®) used by, e.g., Medline, Pubmed, and Cochrane might be the most commonly known¹. Other subject specific databases, such as PsycInfo, have their own controlled vocabulary. The controlled vocabulary helps the searcher by mapping synonyms and concepts that might differ in American English and British English, or where the same concept has been described in different words (Higgins and Green, 2011). Another advantage is the possibility to simultaneously search for more specific terms, ‘explode’ (Higgins and Green, 2011). Author guidelines may require authors to use MeSH-terms to describe the content of their article, rather than keywords chosen by the author, and the National Library of Medicine provides authors with tools for doing this (National Library of Medicine, 2017).

For systematic reviews to fulfill their purpose, methodological quality in all stages are of importance. Discussions on methodology are ongoing; we need for instance to discuss the term “systematic” and how systematic reviews and narrative reviews together can broaden our understanding of a topic (Greenhalgh et al., 2018). Reporting has been found to be inconsistent (Moher et al., 2007; Koffel and Rethlefsen, 2016; Page et al., 2016), so not all systematic reviews are systematic.

The purpose of the present study was to evaluate the quality of non-Cochrane systematic reviews by analyzing their reporting methods, more specifically how the search to identify a body of evidence was performed. This is exemplified by systematic reviews on eye movement desensitization and reprocessing (EMDR), a trauma-focused therapy commonly used for post-traumatic stress disorder (PTSD; Bisson et al., 2013). We carried out a scoping search to map existing research on EMDR, and found a large number of systematic reviews and meta-analyses on PTSD. Cochrane published systematic reviews on psychological therapies for PTSD in adults in 2007 and 2013 (Bisson and Andrew, 2007; Bisson et al., 2013), and for children and adolescents in 2012 and 2016 (Gillies et al., 2012, 2016).

Materials and Methods

We performed a systematic search for studies on EMDR using the following subject term: Eye movement desensitization therapy, and text words: emdr; eye movement desensiti?ation. The search strategy was peer reviewed and adapted to the different databases. A list of the databases and the search strategy is available from http://hdl.handle.net/11250/2597407. The search started on December 12th 2017 and ended on January 25th 2018. In total, 5,576 studies were retrieved. Duplicates (N = 2,723) were removed using EndNote X8 (Clarivate Analytics). Systematic reviews and meta-analyses reporting the effects of EMDR were manually sorted into a separate EndNote-group, after which those covering PTSD in adults were added to a subgroup, resulting in 22 systematic reviews and meta-analyses. We limited the inclusion of systematic reviews and meta-analyses to those published between 2010 and 2017, as we assumed that the authors would be familiar with the PRISMA guidelines, AMSTAR or Cochrane Handbook from 2010. We have excluded one review as we were unable to obtain it in full text (Novo Navarro et al., 2016), and one Cochrane review due to the aim of the study (Bisson et al., 2013). See Figure 1.

FIGURE 1

Figure 1. Prisma2009 flow diagram. Adapted from Moher et al. (2009).

We initially examined all studies independently using the following broad question regarding the literature search from the Norwegian version of the CASP checklist for Systematic reviews (Critical Appraisal, and Skills Programme., 2018): “Do you think all the important, relevant studies were included?”. Response alternatives are Yes, Unclear or No. This question corresponds with that in AMSTAR 2, asking: “Did the review authors use a comprehensive literature search strategy?” (Shea et al., 2017). Secondly, disagreements were solved through discussion. Afterward, the methods chapters were examined and the searches and reporting rated using relevant elements from the Cochrane Handbook (Higgins and Green, 2011) and PRISMA 2009 Checklist (Liberati et al., 2009).

Medline, Embase, and Central are regarded as the most important sources to search for trials (Higgins and Green, 2011). In addition, the Cochrane handbook suggests the inclusion of subject specific databases. PsycInfo and PILOTS (Published International Literature on Traumatic Stress) are relevant due to the research question of the systematic reviews. Reviews stating that these databases were used, were awarded one point for each database. If no database was mentioned the rating was zero.

A comprehensive search should consist of a combination of subject terms and text words (Higgins and Green, 2011), and we assumed that authors of systematic reviews using subject terms would provide precise information on this aspect. We considered the use of subject terms to be a more objective measure, than evaluating the text words used in the search strategy. Reviews that reported using subject terms received one point, reviews not using subject terms received zero points.

We have rated item 1 (Title), item 7 (Information sources), and item 8 (Search) from the PRISMA Checklist. We have also checked whether the reviews include a flow diagram (item 17), and whether or not they refer to the Cochrane review on psychological therapies for PTSD in adults (Bisson and Andrew, 2007; Bisson et al., 2013). These elements are rated with Yes when information on a given criterion is stated, and No when information is lacking, see Table 2.

TABLE 2

Table 2. Elements used for rating search methods and reporting.

Results

The systematic reviews in Table 3 constitute our data sample.

TABLE 3

Table 3. Included systematic reviews.

Eight reviews were published between 2010 and 2013; twelve reviews were published from 2014 to 2017.

Geographically, correspondence address for the included reviews is United States-6, The Netherlands-4, Great Britain-2, Canada-2, Australia-1, Brazil-1, Taiwan-1, China-1, Germany-1, and France-1.

The systematic reviews and meta-analyses were published in sixteen different journals, from ten different publishers.

Initial Screening of Search Methods

The authors of three reviews (Stergiopoulos et al., 2011; Chen et al., 2014; Thomaes et al., 2014) had reported their search strategies in such a manner that we could answer the critical appraisal question: “Do you think all the important, relevant studies were included?” with “Yes”. For four reviews (Zantvoord et al., 2013; Cusack et al., 2016; Swan et al., 2017; Torchalla and Strehlau, 2017), it was unclear whether the reported search strategy had found all important, relevant studies; for two of these, Cusack et al. (2016) and Zantvoord et al. (2013), we were unable to access the full search strategy. We answered the question with “No” for thirteen of the reviews (Lapp et al., 2010; Goodson et al., 2011; Haugen et al., 2012; Ho and Lee, 2012; Mello et al., 2013; Watts et al., 2013; Ehring et al., 2014; Jong et al., 2014; Chen et al., 2015; Haagen et al., 2015; Erford et al., 2016; Lenz et al., 2017; Tribe et al., 2017).

Databases and Subject Terms

Of the three databases (Central, Medline, Embase) recommended for all systematic reviews by the Cochrane handbook (Higgins and Green, 2011), only Medline/Pubmed was used by nineteen reviews. Ten reviews used Embase, and nine Central for identifying single studies. Seventeen reviews used PsycInfo, one review only used PsycInfo out of these databases, while three reviews used Medline, but not PsycInfo. Eleven studies in our sample used PILOTS. Five reviews reported using all five databases, see Figure 2. Four reviews reported using subject terms as part of their search strategy.

FIGURE 2

Figure 2. Number of reviews using selected databases.

The scores for search characteristics in Table 4 are consistent with the findings from the initial screening of search methods, where three reviews were rated Yes, 4 Unclear, and 11 No. All the reviews rated between 1 and 3 were assessed as No, meaning we consider it unlikely that all important and relevant studies were found. The reviews receiving 6 points, were rated Yes [1] and Unclear [1]. One of the reviews scoring 4 points, was also rated yes in the initial screening.

TABLE 4

Table 4. Scores for search characteristics.

The total scores for the search, reporting and context are shown in Table 5.

TABLE 5

Table 5. Included systematic reviews and meta-analysis with results for the search, reporting and context.

Documentation of the Search and Results in Accordance With PRISMA, Figure 3

17 out of 20 reviews identified the paper as either a “meta-analysis” or systematic review’ in the title. All reviews listed the names of the databases used in the search, but only three stated the names of the database providers. Database hosts or full text resources from one publisher were listed as databases in several of the reviews.

FIGURE 3

Figure 3. Use of PRISMA items. Number of articles reviewed adhering to PRISMA guidelines regarding the reporting of Title, Information sources, Search and Flow diagram.

Information on time span, and date on which the search ended was found in 9 and 12 reviews respectively. Time span searched is provided as general information, not for each database. Supplementary search techniques, such as contacting authors and manually searching journals, were used in 17 papers.

The full search strategy from at least one database was reported by five of the reviews.

A total of 14 reviews (70%) used a flow chart. Of these, only three used the PRISMA flowchart, referring to the PRISMA statement.

Context

Seven of the reviews did not refer to the 2007 Cochrane systematic review on psychological therapies for PTSD in adults by Bisson, or the revised version in 2013 (Bisson and Andrew, 2007; Bisson et al., 2013).

Discussion

Designing and conducting a systematic review or a meta-analysis is time-consuming if all the stages from planning to reporting are performed according to appropriate methodological standards (Liberati et al., 2009; Higgins and Green, 2011). The reported search methods can indicate the certainty that the synthesis or analysis is based on an accurate and complete evidence base (Sampson et al., 2009; Cooper et al., 2018).

Our initial screening found that only three out of the twenty systematic reviews or meta-analyses adhered to recommended systematic search methods. This is a general, overall evaluation of the search strategies reported, without applying specific criteria. In her study of 103 systematic reviews from prosthodontic and implant-related journals, Layton found that only 5% met all assessment criteria (Layton, 2017). A recent study of search strategies (Salvador-Olivan et al., 2019) found that 92.7% of 137 systematic reviews published in January 2018 contained strategies with error. It seems likely that our broad estimate of quality regarding the search is higher than that in studies using specific criteria.

High impact journals publish systematic reviews that have been found to be lacking in quality (Koffel and Rethlefsen, 2016), so there is a need for critical appraisal at article level (DORA, 2012).

The Cochrane handbook recommends that all systematic reviews perform searches in Central, Medline and Embase, with Central considered to be the best single database for searching for reports of trials (Higgins and Green, 2011). This recommendation seems to be unfamiliar to many of the authors in our sample. We do not consider the total number of databases to be relevant in our assessment. Many databases may look impressive to the reader, but to enhance comprehensiveness, it is important to search the appropriate databases, thus reducing the risk of database bias (Schlosser et al., 2007; Cooper et al., 2018). Access to appropriate databases can be a challenge if your institution does not subscribe, but PubMed and PILOTS are freely available, and the PILOTS database is central for studies on PTSD. It may be that authors lack knowledge about available and appropriate databases relevant to their research question.

All of the studies in our sample provided information on which databases they used, but only three named the providers of the databases. In contrast, Koffel and Rethlefsen (2016) found that 61% named the database provider. This information is important for the reproducibility of the search strategies; different providers give differing options and functionality for conducting a search (Cooper et al., 2018). Some of the authors seem to be confused about the difference between a database and a provider, stating that they have searched, e.g., Ebscohost and Medline. When only information on the provider is given, it is impossible to infer which databases were searched, or if the search terms were applied across a range of databases.

When giving advice on designing a search strategy, the Cochrane handbook conveys the following summary point: “Avoid too many different search concepts but use a wide variety of synonyms and related terms (both free text and controlled vocabulary terms) combined with ‘OR’ within each concept” (Higgins and Green, 2011). Only four publications in our sample use subject headings. This corresponds with the findings from a study of 300 systematic reviews, where 12% reported using controlled vocabulary terms (Page et al., 2016). It can be questioned whether authors are aware of the increased needs for accuracy and comprehensiveness in the search strategy of a systematic review or meta-analysis and the importance of using subject headings (Sampson et al., 2009; McGowan et al., 2016; Salvador-Olivan et al., 2019), or if they should seek assistance for designing the search strategy. Table 6 illustrates how different text words are interpreted by different databases, and the consequent need for adaptation of search strategies.

TABLE 6

Table 6. Subject specific example of text word searching in two databases.

Research librarians are methodologists and can be included as part of teams conducting reviews (Rethlefsen et al., 2015; Metzendorf and Featherstone, 2018), or librarians can peer review search strategies for systematic reviews, similar to statistics experts (Koffel and Rethlefsen, 2016). The PRESS-guidelines assume that a librarian or an information specialist plans, performs, and reviews a search strategy (Sampson et al., 2009; McGowan et al., 2016). This can have some advantages for the review team, namely time saved through improved search precision, resulting in less studies to examine, and avoidance of late adjustments to the search strategy, and consequently more studies to examine (Sampson et al., 2009). Teams conducting systematic reviews could profit from an inter-disciplinary approach, involving research librarians. We would welcome a discussion on the inclusion of subject terms as a specific criterion for critical appraisal in AMSTAR 2 (Shea et al., 2017), and a more specific reference in item 8 of the PRISMA statement (Liberati et al., 2009).

As far as we can tell, the search criteria stated as a minimum by the Cochrane handbook (Higgins and Green, 2011) are manageable for authors. The discussion on methodology for systematic reviews, and whether we should differentiate between elaborate reviews and ‘leaner’ reviews to cover important clinical questions, is essential (Bastian et al., 2010).

Readers often use titles of publications as a first sifting instrument, thus the title should describe the content of the articles precisely. The PRISMA-guidelines advise including information about the participants, intervention, comparator, outcome, and study design in the title (Liberati et al., 2009). Seventeen of the twenty reviews in our sample included the term ‘systematic review’ or ‘meta-analysis’ in the title. Of the three that failed to do so, one provided this information in the abstract, and the other two in the full text. Page et al. (2016) found that 94% of the publications examined included “systematic review” or “meta-analysis” in the title, increasing from 68.2% for non-Cochrane reviews in 2004 (Moher et al., 2007). Another new study using the PRISMA to assess quality, found that 79 % of systematic reviews include this information (Sharma and Oremus, 2018). We expect a further rise in precise titles, as found by Page, and believe readers will welcome this improvement.

Nine out of the twenty reviews in our sample provided general information on the time span of their search, and twelve out of twenty stated the date on which the search ended. According to the PRISMA-guidelines, information on dates of coverage should be given for each database (Liberati et al., 2009). It is important to distinguish between coverage for each database and time span as general information (Koffel and Rethlefsen, 2016), as databases such as Medline, Embase, and PsycInfo can be searched for different timespans. Page et al. (2016) report that 65% of studies contain information on start and end dates for all databases, while for Cochrane reviews the share is 91%, and in their sample, 29% of the reviews examined gave general information on time span. Koffel and Rethlefsen (2016) found that 25% of the systematic reviews provide start and end dates for each database. Our sample has only general information on included time span for the search, and in addition, we observed a lack of specific information on database providers. This may signal a need for authors to be more conscientious in reporting their methods. Information on timespan for the databases and last date of the search is necessary to reproduce the search strategy and for updating reviews (Liberati et al., 2009).

The majority of reviews we examined added supplementary techniques to their search strategy to support the collection of relevant studies. The most commonly used technique found in other studies was to review reference lists (Koffel and Rethlefsen, 2016; Page et al., 2016), while other techniques can be identifying unpublished trials, or contacts with experts. Among other things, the aim of using supplementary techniques is to reduce the risk of missing relevant studies, and publication bias (Cooper et al., 2018).

The PRISMA guidelines advise that the authors of systematic reviews should include a full search strategy for at least one database (Liberati et al., 2009). Only five of the twenty reviews or meta-analyses in our sample fulfilled this criterion. This is similar to findings by Koffel and Rethlefsen (2016) [24%] and Page et al. (2016) [30%] for non-Cochrane reviews, while Sharma and Oremus (2018) report higher adherence [63.2%]. The main reason for this advice is to make critical evaluation possible and allow replication and updating (Bastian et al., 2010; McGowan et al., 2016), thus giving the reader a chance to decide whether or not the foundations of the work are sound (Sampson et al., 2009).

Similar to Koffel and Rethlefsen (2016), we find that a majority of reviews report their search strategies at a level that is too general, hence insufficient for replication. Item 7 of the PRISMA guideline recommends detailed reporting of information sources, whereas Item 8, Search, can be answered with Yes or No. This can explain why Sharma and Oremus (2018) found high compliance with Item 7 in their study. We argue that adherence to the PRISMA guidelines is necessary to provide sufficient detail to reproduce and assess search strategies.

Flow charts to illustrate identification, screening, eligibility, and inclusion, are used in 14 of the 20 [70%] reviews in our study, although only three of these use the PRISMA flow diagram (Moher et al., 2009). This result corresponds with Page et al. (2016) and Sharma and Oremus (Sharma and Oremus, 2018), 69 and 89.5% respectively. Page et al. differentiate between complete, partial, and no reporting, which can explain the difference.

In their study of systematic reviews from 2012, Koffel and Rethlefsen (2016) ask if enhanced knowledge of reporting and methodology guidelines will improve reporting quality. Our study of systematic reviews and meta-analyses published between 2011 and 2017, using PRISMA and Cochrane Handbook criteria shows no positive development regarding reporting quality. Although some authors, [3/20] in our sample, stated that they followed the PRISMA guidelines, we found that they did not comply with them (Liberati et al., 2009). One explanation can be that editors are more occupied with other details, or lack resources to follow-up on reporting or methodology standards (Moher et al., 2007). Other reasons might be the lack of courses for researchers on systematic search methods and reporting standards (Koffel and Rethlefsen, 2016), or the lack of software to help authors report their systematic review (Page et al., 2016). For the time being, PRISMA checklists do not ensure actual use of the guidelines (Page et al., 2016), and can appear to be a “ticking-the-boxes” exercise (Greenhalgh et al., 2018).

As we noted a large number of reviews and meta-analyses on PTSD in adults in our initial scoping search, we added an extra criterion assessing whether the publications referred to a specific Cochrane review (Bisson and Andrew, 2007; Bisson et al., 2013) addressing this research question. Interestingly, seven of the twenty reviews do not refer to the Bisson review (Bisson and Andrew, 2007; Bisson et al., 2013). It is important that authors of reviews or meta-analyses place their work in the context of other reviews (Bastian et al., 2010). Furthermore, Bastian et al. (2010) point out that this is also the responsibility of editors.

In general, the studies identified in a systematic search form the basis of the review process, and should build a sound base for analyzing and synthesizing evidence (Cooper et al., 2018), and minimize bias in forming conclusions. If the methods used for searching are not systematic or the search strategy is flawed, there is a risk of not including all relevant studies, thus influencing the conclusions of systematic reviews (Salvador-Olivan et al., 2019).

More specifically, poor systematic reviews can affect the field of psychotherapy, and treatments such as EMDR, negatively, as mental health practitioners who wish to find evidence based treatments they can use to treat their patients, may become confused by conflicting statements.

In the present paper, we find that half of the reviews have a score of three points or lower. This result suggests that there is not enough attention to the literature search process. The different phases of planning and doing a systematic review have their own methodological challenges, and may introduce bias in conclusions. There is a possibility that poor quality in the reviews can lead to inaccurate conclusions. We have not focused directly on the conclusions from each of the reviews. That may become the subject of another paper.

There are some weaknesses in this study. Our examination of search methods relies on reported characteristics, and our sample is small. However, assessing systematic reviews and meta-analyses in light of a relatively narrow research question reveals results that raise some important issues regarding the trustworthiness of the body of knowledge pertaining to that question. Furthermore, two reviewers individually screened the methods sections of the reviews and assessed the overall confidence in the completeness of the search. In our evaluation we used the minimum criteria in the Cochrane Handbook (Higgins and Green, 2011) and PRISMA (Liberati et al., 2009) for systematic reviews and meta-analyses, and have discussed our findings with reference to studies with larger samples.

Conclusion

The aim of this article has been to evaluate the quality of non-Cochrane systematic reviews addressing the subject of EMDR as a therapy commonly used for PTSD by analyzing their search and reporting methods. We found inadequacies in the methods employed for searching and reporting the search strategy, which could have been avoided by greater adherence to guiding documents for performing systematic reviews.

The methods used for performing and reporting a search can, like other methods used in a systematic review, enhance or undermine trustworthiness. Review teams could profit from involving different methodologists, such as research librarians. Alternatively, authors should look for, and participate in, courses on systematic searching. Table 7 lists some key points relevant for authors of systematic reviews.

TABLE 7

Table 7. Good practice key points for effective searching and reporting.

Our findings raise important questions for future debate on the risk of omitting studies, thus impairing the conclusions of a systematic review. For clinical purposes, researchers should investigate if, and how, the search strategy in a systematic review affects the body of knowledge and the results. Poor systematic reviews may confuse mental health practitioners who wish to find evidence-based treatments they can use to treat their patients.

Data Availability

The datasets generated for this study are available on request to the corresponding author.

Author Contributions

EO and PA appraised included studies. EO drafted the manuscript. PA, MJ, BA, and KK contributed to critical revision of the draft. All authors conceived and designed the study, and read and approved the final version of the manuscript.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We thank senior adviser Sari Susanna Ormstad, Norwegian Institute of Public Health, for peer reviewing the search strategy.

Footnotes

1. ^ https://www.ncbi.nlm.nih.gov/mesh

References

Agency for healthcare research and quality (2015). Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Available at: https://www.ncbi.nlm.nih.gov/books/NBK47095/ (accessed December 1, 2018).

Google Scholar

Bastian, H., Glasziou, P., and Chalmers, I. (2010). Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 7:e1000326. doi: 10.1371/journal.pmed.1000326

PubMed Abstract | CrossRef Full Text | Google Scholar

Bisson, J. I., and Andrew, M. (2007). Psychological treatment of post-traumatic stress disorder (PTSD).Cochrane Database Syst. Rev. CD003388. doi: 10.1371/journal.pmed.1000326

PubMed Abstract | CrossRef Full Text | Google Scholar

Bisson, J. I., Roberts, N. P., Andrew, M., Cooper, R., and Lewis, C. (2013). Psychological therapies for chronic post-traumatic stress disorder (PTSD) in adults. Cochrane Database Syst. Rev. CD003388. doi: 10.1002/14651858.CD003388.pub4

PubMed Abstract | CrossRef Full Text | Google Scholar

Chen, L., Zhang, G., Hu, M., and Liang, X. (2015). Eye movement desensitization and reprocessing versus cognitive-behavioral therapy for adult posttraumatic stress disorder: systematic review and meta-analysis. J. Nerv. Ment. Dis. 203, 443–451. doi: 10.1097/nmd.0000000000000306

PubMed Abstract | CrossRef Full Text | Google Scholar

Chen, Y. R., Hung, K. W., Tsai, J. C., Chu, H., Chung, M. H., Chen, S. R., et al. (2014). Efficacy of eye-movement desensitization and reprocessing for patients with posttraumatic-stress disorder: a meta-analysis of randomized controlled trials. PLoS One 9:e103676. doi: 10.1371/journal.pone.0103676

PubMed Abstract | CrossRef Full Text | Google Scholar

Cooper, C., Booth, A., Varley-Campbell, J., Britten, N., and Garside, R. (2018). Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med. Res. Methodol. 18:85. doi: 10.1186/s12874-018-0545-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Cooper, H. M. (1982). Scientific guidelines for conducting integrative research reviews. Rev. Educ. Res. 52, 291–302. doi: 10.3102/00346543052002291

CrossRef Full Text | Google Scholar

Critical Appraisal, and Skills Programme. (2018). CASP Checklists (cop. 2018). Available from: https://casp-uk.net/casp-tools-checklists/ (accessed December 1, 2018).

Google Scholar

Cusack, K., Jonas, D. E., Forneris, C. A., Wines, C., Sonis, J., Middleton, J. C., et al. (2016). Psychological treatments for adults with posttraumatic stress disorder: a systematic review and meta-analysis. Clin. Psychol. Rev. 43, 128–141. doi: 10.1016/j.cpr.2015.10.003

PubMed Abstract | CrossRef Full Text | Google Scholar

DORA (2012). San Francisco Declaration on Research Assessment. Available at: https://sfdora.org/read/ (accessed November 15, 2018).

Google Scholar

Ehring, T., Welboren, R., Morina, N., Wicherts, J. M., Freitag, J., and Emmelkamp, P. M. (2014). Meta-analysis of psychological treatments for posttraumatic stress disorder in adult survivors of childhood abuse. Clin. Psychol. Rev. 34, 645–657. doi: 10.1016/j.cpr.2014.10.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Erford, B. T., Gunther, C., Duncan, K., Bardhoshi, G., Dummett, B., Kraft, J., et al. (2016). Meta-analysis of counseling outcomes for the treatment of posttraumatic stress disorder. J. Counsel. Dev. 94, 13–30. doi: 10.1002/jcad.12058

CrossRef Full Text | Google Scholar

Foster, M. J., and Jewell, S. T. (2017). Assembling the Pieces 0f a Systematic Review: a Guide for Librarians. Lanham, MD: Rowman & Littlefield.

Google Scholar

Gillies, D., Maiocchi, L., Bhandari, A. P., Taylor, F., Gray, C., and O’Brien, L. (2016). Psychological therapies for children and adolescents exposed to trauma. Cochrane Database Syst. Rev. 10:CD012371. doi: 10.1002/14651858.CD012371

PubMed Abstract | CrossRef Full Text | Google Scholar

Gillies, D., Taylor, F., Gray, C., O’Brien, L., and D’Abrew, N. (2012). Psychological therapies for the treatment of post-traumatic stress disorder in children and adolescents. Cochrane Database Syst. Rev. 12:CD006726. doi: 10.1002/14651858.CD006726.pub2

PubMed Abstract | CrossRef Full Text | Google Scholar

Goodson, J., Helstrom, A., Halpern, J. M., Ferenschak, M. P., Gillihan, S. J., and Powers, M. B. (2011). Treatment of posttraumatic stress disorder in U.S. combat veterans: a meta-analytic review. Psychol. Rep. 109, 573–599. doi: 10.2466/02.09.15.16.PR0.109.5.573-599

PubMed Abstract | CrossRef Full Text | Google Scholar

Greenhalgh, T., Thorne, S., and Malterud, K. (2018). Time to challenge the spurious hierarchy of systematic over narrative reviews? Eur. J. Clin. Invest. 48:e12931. doi: 10.1111/eci.12931

PubMed Abstract | CrossRef Full Text | Google Scholar

Haagen, J. F., Smid, G. E., Knipscheer, J. W., and Kleber, R. J. (2015). The efficacy of recommended treatments for veterans with PTSD: a metaregression analysis. Clin. Psychol. Rev. 40, 184–194. doi: 10.1016/j.cpr.2015.06.008

PubMed Abstract | CrossRef Full Text | Google Scholar

Haugen, P. T., Evces, M., and Weiss, D. S. (2012). Treating posttraumatic stress disorder in first responders: a systematic review. Clin. Psychol. Rev. 32, 370–380. doi: 10.1016/j.cpr.2012.04.001

PubMed Abstract | CrossRef Full Text | Google Scholar

Higgins, J., and Green, S. (2011). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. Available from: www.handbook.cochrane.org (accessed February 12, 2019).

Google Scholar

Ho, M., and Lee, C. (2012). Cognitive behaviour therapy versus eye movement desensitization and reprocessing for post-traumatic disorder-Is it all in the homework then? Eur. Rev. Appl. Psychol. 62, 253–260. doi: 10.1016/j.erap.2012.08.001

CrossRef Full Text | Google Scholar

Institute of medicine (2011). Finding What Works in Health Care: Standards for Systematic Reviews. Available at: http://www.nationalacademies.org/hmd/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx (accessed December 1, 2018).

Google Scholar

Ioannidis, J. P. (2016). The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 94, 485–514. doi: 10.1111/1468-0009.12210

PubMed Abstract | CrossRef Full Text | Google Scholar

Jong, K. d., Knipscheer, J. W., Ford, N., and Kleber, R. J. (2014). The efficacy of psychosocial interventions for adults in contexts of ongoing man-made violence—a systematic review. Health 6:504. doi: 10.4236/health.2014.66070

CrossRef Full Text | Google Scholar

Koffel, J. B., and Rethlefsen, M. L. (2016). Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study. PLoS One 11:e0163309. doi: 10.1371/journal.pone.0163309

PubMed Abstract | CrossRef Full Text | Google Scholar

Lapp, L. K., Agbokou, C., Peretti, C. S., and Ferreri, F. (2010). Management of post traumatic stress disorder after childbirth: a review. J. Psychosom. Obstet. Gynecol. 31, 113–122. doi: 10.3109/0167482X.2010.503330

PubMed Abstract | CrossRef Full Text | Google Scholar

Layton, D. (2017). A critical review of search strategies used in recent systematic reviews published in selected prosthodontic and implant-related journals: are systematic reviews actually systematic? Int. J. Prosthodont. 30, 13–21. doi: 10.11607/ijp.5193

PubMed Abstract | CrossRef Full Text | Google Scholar

Lenz, A. S., Haktanir, A., and Callender, K. (2017). Meta-analysis of trauma-focused therapies for treating the symptoms of posttraumatic stress disorder. J. Couns. Dev. 95, 339–353. doi: 10.1002/jcad.12148

CrossRef Full Text | Google Scholar

Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P., et al. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 6:e1000100. doi: 10.1371/journal.pmed.1000100

PubMed Abstract | CrossRef Full Text | Google Scholar

McGowan, J., Sampson, M., Salzwedel, D. M., Cogo, E., Foerster, V., and Lefebvre, C. (2016). PRESS peer review of electronic search strategies: 2015 guideline statement. J. Clin. Epidemiol. 75, 40–46. doi: 10.1016/j.jclinepi.2016.01.021

PubMed Abstract | CrossRef Full Text | Google Scholar

Mello, P., Silva, G., Donat, J., and Kristensen, C. (2013). An update on the efficacy of cognitive-behavioral therapy, cognitive therapy, and exposure therapy for posttraumatic stress disorder. Int. J. Psychiatry Med. 46, 339–357. doi: 10.2190/PM.46.4.b

PubMed Abstract | CrossRef Full Text | Google Scholar

Metzendorf, M. I., and Featherstone, R. M. (2018). Ensuring quality as the basis of evidence synthesis: leveraging information specialists’ knowledge, skills, and expertise. Cochrane Database Syst. Rev. 4:ED000125. doi: 10.1002/14651858.ED000125

PubMed Abstract | CrossRef Full Text | Google Scholar

Moher, D., Liberati, A., Tetzlaff, J., and Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann. Intern. Med. 151, 264–269.

Google Scholar

Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., and Altman, D. G. (2007). Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 4:e78. doi: 10.1371/journal.pmed.0040078

PubMed Abstract | CrossRef Full Text | Google Scholar

Morton, S. C., Murad, M. H., O’Connor, E., Lee, C. S., Booth, M., Vandermeer, B. W., et al. (2018). Quantitative Synthesis—An Update: Methods Guide for Comparative Effectiveness Reviews. Rockville, MD: Agency for Healthcare Research and Quality.

Google Scholar

National Library of Medicine (2017). Suggestions for Finding Author Keywords Using MeSH Tools. Available from: https://www.nlm.nih.gov/mesh/authors.html (accessed December 4, 2018).

Google Scholar

Novo Navarro, P., Landin-Romero, R., Guardiola-Wanden-Berghe, R., Moreno-Alcazar, A., Valiente-Gomez, A., Lupo, W., et al. (2016). 25 years of eye movement desensitization and reprocessing (EMDR): the EMDR therapy protocol, hypotheses of its mechanism of action and a systematic review of its efficacy in the treatment of post-traumatic stress disorder. Rev. Psiquiatr. Salud Ment. 11, 101–114. doi: 10.1016/j.rpsm.2015.12.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Page, M. J., Shamseer, L., Altman, D. G., Tetzlaff, J., Sampson, M., Tricco, A. C., et al. (2016). Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 13:e1002028. doi: 10.1371/journal.pmed.1002028

PubMed Abstract | CrossRef Full Text | Google Scholar

PLoS Medicine Editors (2007). Many reviews are systematic but some are more transparent and completely reported than others. PLoS Med. 4:e147. doi: 10.1371/journal.pmed.0040147

PubMed Abstract | CrossRef Full Text | Google Scholar

Rethlefsen, M. L., Farrell, A. M., Trzasko, L. C. O., and Brigham, T. J. (2015). Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J. Clin. Epidemiol. 68, 617–626. doi: 10.1016/j.jclinepi.2014.11.025

PubMed Abstract | CrossRef Full Text | Google Scholar

Salvador-Olivan, J. A., Marco-Cuenca, G., and Arquero-Aviles, R. (2019). Errors in search strategies used in systematic reviews and their effects on information retrieval. J. Med. Libr. Assoc. 107, 210–221. doi: 10.5195/jmla.2019.567

PubMed Abstract | CrossRef Full Text | Google Scholar

Sampson, M., McGowan, J., Cogo, E., Grimshaw, J., Moher, D., and Lefebvre, C. (2009). An evidence-based practice guideline for the peer review of electronic search strategies. J. Clin. Epidemiol. 62, 944–952. doi: 10.1016/j.jclinepi.2008.10.012

PubMed Abstract | CrossRef Full Text | Google Scholar

Schlosser, R. W., Wendt, O., and Sigafoos, J. (2007). Not all systematic reviews are created equal: considerations for appraisal. Evid. Based. Commun. Assess. Interv. 1, 138–150. doi: 10.1080/17489530701560831

CrossRef Full Text | Google Scholar

Sharma, S., and Oremus, M. (2018). PRISMA and AMSTAR show systematic reviews on health literacy and cancer screening are of good quality. J. Clin. Epidemiol. 99, 123–131. doi: 10.1016/j.jclinepi.2018.03.012

PubMed Abstract | CrossRef Full Text | Google Scholar

Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., et al. (2017). AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 358:j4008. doi: 10.1136/bmj.j4008

PubMed Abstract | CrossRef Full Text | Google Scholar

Stergiopoulos, E., Cimo, A., Cheng, C., Bonato, S., and Dewa, C. S. (2011). Interventions to improve work outcomes in work-related PTSD: a systematic review. BMC Public Health 11:838. doi: 10.1186/1471-2458-11-838

PubMed Abstract | CrossRef Full Text | Google Scholar

Swan, S., Keen, N., Reynolds, N., and Onwumere, J. (2017). Psychological interventions for post-traumatic stress symptoms in psychosis: a systematic review of outcomes. Front. Psychol. 8:341. doi: 10.3389/fpsyg.2017.00341

PubMed Abstract | CrossRef Full Text | Google Scholar

Thomaes, K., Dorrepaal, E., Draijer, N., Jansma, E. P., Veltman, D. J., and van Balkom, A. J. (2014). Can pharmacological and psychological treatment change brain structure and function in PTSD? A systematic review. J. Psychiatr. Res. 50, 1–15. doi: 10.1016/j.jpsychires.2013.11.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Torchalla, I., and Strehlau, V. (2017). The evidence base for interventions targeting individuals with work-related PTSD: a systematic review and recommendations. Behav. Modif. 42, 273–303. doi: 10.1177/0145445517725048

PubMed Abstract | CrossRef Full Text | Google Scholar

Tribe, R. H., Sendt, K. V., and Tracy, D. K. (2017). A systematic review of psychosocial interventions for adult refugees and asylum seekers. J. Ment. Health. doi: 10.1080/09638237.2017.1322182 [Epub ahead of print].

PubMed Abstract | CrossRef Full Text | Google Scholar

Watts, B. V., Schnurr, P. P., Mayo, L., Young-Xu, Y., Weeks, W. B., and Friedman, M. J. (2013). Meta-analysis of the efficacy of treatments for posttraumatic stress disorder. J. Clin. Psychiatry 74, e551–e557. doi: 10.4088/JCP.12r08225

PubMed Abstract | CrossRef Full Text | Google Scholar

Wormald, R., and Evans, J. (2018). What makes systematic reviews systematic and why are they the highest level of evidence? Ophthalmic Epidemiol. 25, 27–30. doi: 10.1080/09286586.2017.1337913

PubMed Abstract | CrossRef Full Text | Google Scholar

Zantvoord, J. B., Diehle, J., and Lindauer, R. J. (2013). Using neurobiological measures to predict and assess treatment outcome of psychotherapy in posttraumatic stress disorder: systematic review. Psychother. Psychos. 82, 142–151. doi: 10.1159/000343258

PubMed Abstract | CrossRef Full Text | Google Scholar