Jarkko Jalava
Stephanie Griffiths
Rasmus Rosenberg Larsen
B. Emma Alcott- 1Department of Interdisciplinary Studies, Okanagan College, Penticton, BC, Canada
- 2Department of Psychology, Okanagan College, Penticton, BC, Canada
- 3Werklund School of Education, University of Calgary, Calgary, AB, Canada
- 4Forensic Science Program and Department of Philosophy, University of Toronto Mississauga, Mississauga, ON, Canada
- 5Irving K. Barber School of Arts and Sciences, University of British Columbia, Kelowna, BC, Canada
Questionable research practices are a well-recognized problem in psychology. Coding bias, or the tendency of review studies to disproportionately cite positive findings from original research, has received comparatively little attention. Coding bias is more likely to occur when original research, such as neuroimaging, includes large numbers of effects, and is most concerning in applied contexts. We evaluated coding bias in reviews of structural magnetic resonance imaging (sMRI) studies of PCL-R psychopathy. We used PRISMA guidelines to locate all relevant original sMRI studies and reviews. The proportion of null-findings cited in reviews was significantly lower than those reported in original research, indicating coding bias. Coding bias was not affected by publication date or review design. Reviews recommending forensic applications—such as treatment amenability or reduced criminal responsibility—were no more accurate than purely theoretical reviews. Coding bias may have contributed to a perception that structural brain abnormalities in psychopaths are more consistent than they actually are, and by extension that sMRI findings are suitable for forensic application. We discuss possible sources for the pervasive coding bias we observed, and we provide recommendations to counteract this bias in review studies. Until coding bias is addressed, we argue that this literature should not inform conclusions about psychopaths' neurobiology, especially in forensic contexts.
Introduction
Psychopathy, as assessed by the Hare Psychopathy Checklist-Revised (PCL-R), is a psychiatric construct associated with affective and interpersonal abnormalities as well as antisocial behavior (Hare, 2003). In the criminal justice system, PCL-R evaluations have been used to inform decisions about such things as sentencing, institutional placement, parole, juvenile transfers, and treatment amenability (Gacono, 2016; Patrick, 2018). Neuroimaging studies have found structural and functional abnormalities in psychopaths, and as a result many researchers view psychopathy as a neurobiological disorder (e.g., Blair, 2013; Lushing et al., 2016; Sethi et al., 2018; Yang and Raine, 2018). Some authors have argued that these abnormalities might be taken into account when determining psychopaths' criminal responsibility (e.g., Blair, 2008; Anderson and Kiehl, 2013; Raine, 2019), amenability to neurosurgery or pharmacological treatment (e.g., De Ridder et al., 2009; Glenn and Raine, 2009), and when trying to predict their future dangerousness (e.g., Nadelhoffer et al., 2012; Umbach et al., 2015). Neuroimaging evidence on psychopaths has already been presented in court, including in death penalty hearings [e.g., State v. Brian, 2009; State v. Jerome, 2015; see also Denno (2015)].
However, the reliability of psychological data—and by extension their readiness for application—has come under increasing scrutiny. For decades, psychological research has been criticized for producing an unrealistically high proportion of positive findings (Sterling, 1959; Greenwald, 1975; Sterling et al., 1995; Fanelli, 2012). Recent studies describe a particularly acute problem in cognitive neuroscience where, depending on the year, up to 90% of all published findings have been positive (Fanelli, 2012). The high prevalence of positive findings is concerning for two reasons. First, as neither neuroimaging methods nor psychological tests have particularly high reliability, a significant proportion of reported findings may be false positives. Second, there is a well-recognized set of biases favoring positive findings, which may be eliminating true null-findings from the literature (Vul et al., 2009; Wager et al., 2009; Button et al., 2013; Nugent et al., 2013; Szucs and Ioannidis, 2017; Vul and Pashler, 2017). The biases toward positive findings include the file drawer problem (only studies with positive findings are submitted to journals; Rosenthal, 1979), publication and reporting bias (journals are more likely to publish and authors to report positive than null-findings; Jennings and Van Horn, 2012; David et al., 2013, 2018; Dwan et al., 2013; Ioannidis et al., 2014), and p-hacking (researchers use flexible data analyses to produce positive finding; Nelson et al., 2018).
These and other Questionable Research Practices [QRPs; see John et al. (2012)] in original research may also skew review studies and meta-analyses. A recent study comparing effects from meta-analyses and large-scale replication studies in psychology—the latter avoiding QRPs through pre-registration—found that meta-analytic effect sizes were indeed significantly larger (Kvarven et al., 2020). Some researchers have argued that reviews and meta-analyses may actually amplify biases in original research. This could be for at least two reasons. First, since biases in original studies tend to be systematic—toward fewer nulls—aggregating the studies in meta-analyses will only intensify the biases (Nelson et al., 2018). Second, reviews and meta-analyses (henceforth, “review literature”) may have QRPs of their own. These include funding bias (e.g., Jørgensen et al., 2006; Bes-Rastrollo et al., 2013; Mandrioli et al., 2016), citation bias [Duyx et al., 2017; but not always; see Nuijten et al. (2020)], spin (e.g., Drucker et al., 2016; Yavchitz et al., 2016; McGrath et al., 2017), and post-hoc changes to registered review protocols (Silagy et al., 2002).
An additional QRP in review literature that has received far less attention is the so-called coding bias (also known as data extraction bias; Petticrew and Roberts, 2008). Coding bias refers to the decisions reviewers make about which data to extract from a study. Coding bias in review literature is analogous to reporting bias in original research—just as an original study can highlight positive findings in abstracts while burying nulls in supplemental tables or not reporting them at all, a reviewer can do the same by selectively coding positive findings (coding bias is different from citation bias, as the latter only addresses biases in study choice, not in within-study effects). Coding bias is most likely to occur in fields such as cognitive neuroscience where a single study can report a large number of effects, and where reviewers therefore enjoy many of the same kinds of “degrees of freedom” as original researchers (Müller et al., 2018). Although coding bias has received some attention (Orwin and Cordray, 1985; Wortman and Bryant, 1985; Petticrew and Roberts, 2008), it has not been systematically evaluated. In this paper, we examine coding bias in neuroimaging research on psychopathy. We define coding bias as a selective extraction of positive findings from original studies by authors of review literature. To measure coding bias, we compared the proportion of null-findings in meta-analyses and review studies to the proportion of null-findings in original research. We adopted “Preferred Reporting Items for Systematic Reviews and Meta-Analyses” (PRISMA) guidelines for locating original studies and review literature (Liberati et al., 2009), and we used an expert consensus extraction strategy for all effects. We also examined whether the agreement between original and review literatures varied as a function of publication date or type of review.
We focused on two clearly defined parameters: psychopathy measured by the Hare Psychopathy Checklist-Revised (PCL-R) or its Screening Version (PCL:SV) (Hart et al., 1995; Hare, 2003) and brain abnormalities as described by structural magnetic resonance imaging (sMRI) data. The PCL-R is considered the standard measure for psychopathy in forensic settings (Hare, 2003, 2016; Glenn and Raine, 2008). We focused on sMRI as opposed to functional (fMRI) studies, as fMRI studies employ a wide range of tasks that make between-study comparisons difficult. Also, review studies often fail to include descriptions of task conditions in their summaries of fMRI findings, making it difficult to know exactly which task a reviewer is referring to.
Methods
Literature Search Inclusion and Exclusion Criteria
Original sMRI Studies
Studies were included if they reported either case-control or correlational sMRI data on PCL-R or PCL:SV defined psychopathy samples. Exclusion criteria were (i) studies published in language other than English, (ii) studies conducted on youth or adolescents, and (iii) studies that did not report sufficient detail on PCL-R or PCL:SV scores (e.g., not reporting total scores).
Review Literature
Meta-analyses and review studies were included if their stated or implied purpose was to review neuroimaging research on psychopathy, and included sMRI data on PCL-R or PCL:SV defined psychopathy. Exclusion criteria were (i) studies published in language other than English, (ii) studies reviewing data only on youth or adolescents, (iii) studies that did not report sufficient detail on PCL-R or PCL:SV scores (e.g., not reporting total scores), (iv) studies published by any of the current authors to avoid the possibility of bias.
Search Strategy
Original sMRI Studies
We conducted a full-text, English-language only PRISMA search in the years 1995–2020, using the keyword sets (Psychopathic OR Psychopathy OR psychopath OR pcl*) AND (neuro* OR brain) AND (smri OR structural). The initial search yielded 274 records (Medline n = 124; PsycINFO n = 150). We excluded 184 records that were duplicates and/or thematically irrelevant (i.e., the keywords or titles clearly suggested the article was unrelated to our search topic). The identified 90 records were exported to Endnote X9 (Clarivate Analytics), where we scanned the titles and abstracts to determine their relevance. At this step we excluded 55 articles. We then examined the full text for the remaining articles, and excluded 15 studies that either (a) used unrelated design, (b) used a measure other than the PCL-R, (c) did not report PCL-R total score, or (d) were unpublished.1 Twenty records were retained for analysis in our study (for workflow, see Figure 1). Finally, we manually scanned recent neuroimaging review studies on psychopathy to determine if our initial search missed any relevant publications. This manual scan identified an additional 18 records, resulting in a total of 38 studies retained for our analysis. We excluded unpublished studies, even when cited in review literature, as they were not available for coding.
Figure 1. PRISMA flow charts.
Review Literature
We conducted a full-text, English-language only PRISMA search in the years 1995–2020, using the keyword sets (Psychopathic OR Psychopathy OR psychopath OR pcl*) AND (neuro* OR brain) AND (meta* OR review). The initial search yielded 825 records (Medline n = 534; PsycINFO n = 291). We excluded 712 records that were duplicates and/or thematically irrelevant (i.e., the keywords or titles clearly suggested the article was unrelated to our search topic). We retained 113 records, which we exported to Endnote X9 (Clarivate Analytics), where we scanned the titles and abstracts to determine their relevance. At this step we excluded 74 articles. We then examined the full text for the remaining 39 articles, and excluded six studies that either (a) did not disclose sufficient information on the studies reviewed, (b) did not use the PCL-R, (c) did not use sMRI, or (d) could not be located.2 Thirty three records were retained for analysis in our study (for workflow, see Figure 1). Finally, we conducted a manual scan (reference sections of review studies and recommendations from manuscript reviewers) to determine if our initial search missed any relevant publications. This manual scan identified an additional 12 relevant studies, resulting in a total of 45 studies retained for our analysis.
Data Coding
Null-Findings in Original sMRI Studies
We followed a systematic coding strategy for null- and positive findings [see Griffiths and Jalava (2017)]. We first examined the percentage of null-findings in the original 38 research studies by recording all regions of interest (ROIs) identified in the introduction section of each article. If statistically significant regions beyond the ROIs were reported in results section, these regions were added to the total ROIs. We then recorded all ROIs in the results section by examining test statistics and/or p-values to identify statistically significant findings. Null-findings were identified either by test statistics and/or p-values or by missing results for ROIs that had been clearly identified in the introduction. In case of whole-brain analysis/exploratory research supplementary tables were used to identify null-rejections and null-findings. We followed reporting patterns of original studies with each reported effect counting as an ROI. In between-group designs, only group comparisons were reported (i.e., no correlations between foci and PCL-R score). In research designs using more than two groups, all group comparisons were recorded. Psychopathy groups included any subjects indicated as psychopathic (e.g., “medium psychopathy” and “high psychopathy” or “successful psychopath” and “unsuccessful psychopath”). We did not report on regions recorded as manipulation checks or methodological controls. White and gray matter and lateralized findings were included as separate data points. Finally, when relevant, only corrected findings were reported [e.g., controlling for multiple comparisons, small volumes, and drug use; see Müller et al. (2018)]. Two of the authors (J.J., S.G.) reviewed the number of foci. Any disagreements were resolved by a third author (R.R.L.).
Null-Findings in Review Literature
We adopted the following coding strategy for the 45 studies included: We examined the number of foci described either as a positive finding (increased or decreased volume, abnormal shape, etc.) or null-finding for PCL-R or PCL:SV total scores or for all factors. We included findings only when a clear comparison (e.g., psychopaths vs. control) or correlations with PCL-R or PCL:SV score in specified regions was reported. White and gray matter, and right and left findings were each scored separately. The same approach was used for different structural measures (e.g., volume, thickness, etc.). If a finding was described as “bilateral” or referred to in plural (e.g., amygdalae, gyri, nuclei, etc.), it was coded accordingly as two separate findings. If a finding in an individual study referred to “volumes” (e.g., amygdala volumes), it was coded as two separate findings. However, if “volumes” referred to more than one study, it was coded as one finding per study. Three of the authors (J.J., S.G., and E.A.) reviewed the number of foci. Any disagreements were resolved in a fourth review by two of the authors (J.J. and S.G.). For more details on the coding process and examples of it, see Appendix in Supplementary Material.
Results
Proportion of Null-Findings in Original sMRI Studies
The above method yielded the following ratios: Of the 791 effects recorded in 38 original sMRI studies 64.10% (507 out of 791) were null-findings, and 35.90% (284 out of 791) were positive findings (see Table 1). We examined the data for outliers, and identified one study (Boccardi et al., 2011) that reported a total of 312 comparisons, out of which 130 were positive findings. When we excluded these 312 comparisons, the proportion of null-findings across the remaining 37 studies was 67.85% (325 out of 479), indicating that the single study with a large number of comparisons did not unduly affect the proportion of null-findings.
Table 1. Summary of sMRI effects.
Proportion of Null-Findings in Review Literature
We included 45 relevant publications, of which 43 were review studies and two were meta-analyses. Overall, of the 1,001 effects reported in the review literature, 8.99% (N = 90) were null-findings. The remaining 91.01% (N = 911) were positive findings (see Table 2). The difference between the proportion of null-findings in original studies and review literature was statistically significant (χ2 = 1321.07, p < 0.00001).
Table 2. Summary of review study effects.
In order to exclude the possibility that something other than coding bias can explain the discrepancy, we considered the possibility that reviews focused on theoretically important regions could have yielded more positive findings than theoretically peripheral areas. We ran two additional analyses: First, to account for the possibility that a disproportionate number of null-findings came from exploratory, whole-brain analyses of theoretically unrelated regions, we repeated the analysis of the sMRI research excluding studies whose authors identified them as exploratory (these studies were Müller et al., 2008; Tiihonen et al., 2008; Howner et al., 2012; Contreras-Rodríguez et al., 2015). This analysis yielded 67.25% (N = 460) null-findings. In other words, the proportion of null-findings did not appear to be driven by exploratory studies reporting on areas not theorized to be relevant to psychopathy.
Second, we reviewed citation patterns at the effect level. We focused on the amygdala, because it is (a) central to prevailing neurobiological theories of psychopathy and thus widely cited in the review literature (Kiehl, 2006; Blair, 2008), and (b) narrowly and consistently defined across original and review literature, permitting a direct focal comparison between the two types of literatures. The original sMRI studies reported 13 results for the amygdala: six null-findings (Schiltz et al., 2007; de Oliveira-Souza et al., 2008; Tiihonen et al., 2008; Cope et al., 2012; Ermer et al., 2012; Gregory et al., 2012), four volumetric reductions (Yang et al., 2009b, 2010; Ermer et al., 2012; Contreras-Rodríguez et al., 2015, one enlargement (Boccardi et al., 2011), one non-linear PCL-R and volume correlation (Schiffer et al., 2011), and one difference in surface shape (Yang et al., 2010).3 The percentage of null-findings thus accounted for 46.15% of the findings. In contrast, review studies reported 116 findings for the amygdala, of which three (2.59%) were null-findings. Therefore, low proportions of null-findings cannot be attributed to reviewers documenting positive findings in theoretically salient regions and ignoring peripheral noise in the sMRI literature.
To account for the possibility that some reviewers might report fewer null-findings simply because the prevalence of null-findings has changed over time (i.e., perhaps earlier original research reported fewer null-findings than later research), we examined the proportion of null-findings in the original sMRI studies at 5 year intervals. As is apparent in Figure 2 and Table 3, the proportion of nulls has decreased with time in both original studies and review literature [the trend appears similar to that in neuroscientific literature in general; see Fanelli (2012)]. Therefore, chronological changes or study availability do not appear to explain our results.
Figure 2. Null sMRI findings reported over time.
Table 3. Null sMRI findings reported over time.
Finally, to ensure that we did not include original studies that reviewers had designated as irrelevant we compared our list of original studies to studies cited in the review literature. All the original studies in our list were cited at least once in the review literature (see Table 2).
Proportion of Null-Findings by Review Type
We classified reviews according to their study design. We identified two meta-analyses, three systematic reviews, 27 narrative reviews, 12 targeted/focused reviews4, and one editorial. The meta-analyses included only five effects and the editorial included two. The remaining three categories included much larger number of effects: systematic reviews (n =185), narrative reviews (n = 636), and targeted reviews (n = 175). We examined whether reviews using a systematic search strategy reported more null-findings than reviews using other designs. Narrative reviews were more likely to cite null-findings than systematic or targeted reviews [χ2 (2, N = 996) = 14.87, p < 0.001]. However, the difference was entirely driven by a single narrative review (Plodowski et al., 2009) that cited an accurate proportion of null-findings. After removing this outlier, there was no difference in the proportion of nulls by review type [χ2 (2, N = 957) = 2.15, p = ns]. That is, reviews using systematic database searches were no less subject to coding bias than other types of reviews.
We also classified reviews into those that made forensic recommendations (applied reviews) and those that did not (theoretical reviews). Theoretical reviews (N = 30) reported 669 effects, of which 10.91% (N = 73) were null-findings and 89.09% (N = 596) were positive findings. Applied reviews (N = 15) reported 334 effects, of which 5.39% (N = 18) were null-findings and 94.61% (N = 316) were positive findings. The difference between applied and theoretical reviews was statistically significant (χ2 = 10.47, p < 0.01). One outlier (Plodowski et al., 2009), however, reported 24.7% (N = 18) of the 73 null-findings in theoretical reviews. After removing this outlier, the discrepancy was no longer significant (χ2 = 2.84, p = ns).
Finally, we examined the proportion of review studies that found support for neurobiological bases of psychopathy. Twenty-one of the 30 theoretical reviews (70%) found general support for the neurobiological bases of psychopathy while four studies found the data to be inconclusive (Herba et al., 2007; Muller, 2010; Koenigs et al., 2011; Pujara and Koenigs, 2014). One meta-analysis (Yang and Raine, 2009) examined whether PCL-R scores moderated the relationship between antisocial behavior and prefrontal volumes, and found that they did not. The studies that found the data to be inconclusive did so based on (a) the widespread nature of the findings and/or (b) the fact that the positive findings included both increased and decreased volume. Three studies reached only tentative conclusions (Plodowski et al., 2009; Wahlund and Kristiansson, 2009; Santana, 2016), and one meta-analysis (Aoki et al., 2014) did not report findings on psychopathy separately from general antisocial traits and behaviors. In contrast, all 15 applied reviews interpreted the data to indicate neurobiological bases of psychopathy.
Discussion
Neurobiological reviews of PCL-R and PCL:SV psychopathy significantly under-report null-findings in sMRI research, indicating widespread coding bias. The majority (64.18%) of original sMRI findings were nulls, whereas nulls made up a small minority (8.99%) of effects in review literature. Reviewers, in other words, preferentially reported data supporting neurobiological models of psychopathy. We found no evidence that the reporting imbalance was due to factors other than bias: systematic, narrative, and targeted reviews all reported disproportionately few nulls (though meta-analyses reported too few effects to evaluate), the pattern was stable across time, and not driven by exploratory research or outliers. Notably, reviews calling for forensic application of the data, such as treatment, criminal responsibility, punishment, and crime prediction, were no more accurate than purely theoretical reviews. Applied reviews were, however, more likely than theoretical reviews to conclude that the data supported neurobiological bases of psychopathy. These findings are surprising, as applied reviews in other fields—such as those examining drug safety and efficacy—typically face the highest burden of proof and are thus most likely to emphasize limitations in the data [see e.g., Köhler et al. (2015)].
Our study is the first to systematically examine coding bias in cognitive neuroscience. Although our findings are limited to structural imaging in psychopathy, they suggest that coding bias should be considered alongside more widely recognized Questionable Research Practices (QRPs) such as p-hacking, reporting bias, publication bias, citation bias, and the file drawer problem. QRPs in original research filter out null-findings at early stages of the research and publication process, while coding and citation bias further distort the state of scientific knowledge by eliminating null findings from reviews. In addition to coding bias, we found evidence of reporting bias during our review of sMRI studies. Null-findings in the original literature were rarely reported in the study abstracts and were frequently not reported fully in results sections. Nulls often appeared only in data or supplemental tables, and in some cases they had to be inferred by examining ROIs mentioned in the introduction but not in the results section. This illustrates how QRPs are not mutually exclusive, and the presence of one QRP may also signal the presence of another [see e.g., Agnoli et al. (2017)].
The coding bias we observed may have a number of explanations. First, reviewers may have been subject to confirmation bias. Confirmation bias refers to the tendency to weigh evidence that confirms a belief more heavily than evidence that does not (Nickerson, 1998). Reviewers in our study may have assumed neurobiological abnormalities in psychopaths—perhaps from previous reviews—and looked more carefully for data to confirm that assumption. Confirmation bias has been cited as a possible explanation for under-reporting of null-findings in original research (Forstmeier et al., 2017). Our findings suggest that it may play a role in review literature, where null-findings would be especially difficult to square with theories presuming group differences [see e.g., Sterling et al. (1995) and Ferguson and Heene (2012)], and reporting bias would make it very hard to locate disconfirming (null) findings. Second, reviewers may have been following convention. The earliest review studies did not generally include null-findings, and later reviews may have interpreted this as a precedent to follow. Third, explicit and tacit publication preferences may increase coding bias. Research tracking original studies from grant proposal to publication show that most null-findings are not even written up for publication, and that journals—particularly top-tier journals—show a marked preference for strong positive findings (Franco et al., 2014; Ioannidis et al., 2014). Similarly, review authors may have declined to submit reviews with inconclusive findings. Given the extent of publication bias, it is also possible that journal editors may have been more likely to reject inconclusive reviews in favor of those summarizing consistent, positive findings.
Coding bias observed in our study has a number of potential effects. Aside from distorting the true state of knowledge about structural brain abnormalities in psychopaths, it may also have led at least some researchers and courts to believe that the abnormalities are consistent enough for forensic application. This may have encouraged practitioners to de-emphasize or overlook more reliable, behavioral indicators of criminal responsibility, future dangerousness and treatment amenability in favor of less reliable predictors, such as brain structure. Neuroprediction of crime has a number of empirical shortcomings, such as unknown measurement error and inadequate outcome variables (Poldrack et al., 2018). Using MRI data to predict crime can thus introduce substantial error into an already imperfect process (e.g., Douglas et al., 2017). Neurobiologically-informed assessments and treatments are even less likely to be effective if the population's neurobiology is fundamentally misunderstood. Given the extent of coding bias in the psychopathy literature, such interventions may in fact be harmful.
More broadly, coding bias may have contributed to reverse inference [see Scarpazza et al. (2018)] whereby reports of brain abnormalities are taken as proof that psychopathy is a legitimate diagnostic category [for an argument such as this, see e.g., Kiehl and Hoffman (2011)].5 Similarly, some researchers have suggested that psychopathy diagnoses could be enhanced by neuroimaging evidence (e.g., Hulbert and Adeli, 2015). Arguments of this sort can detract from problems in other aspects of the PCL-R, particularly in its psychometric properties. Recently, these critiques have intensified, with authors raising concerns about the reliability of the PCL-R, its utility in forensic contexts (DeMatteo et al., 2020), its factor structure, and its predictive validity (Boduszek and Debowska, 2016). Using neurobiology to validate psychopathy as a diagnostic category is doubly problematic: not only are presumed brain abnormalities in psychopathy broad and non-specific [for problems in reverse inference, see Poldrack (2011) and Scarpazza et al. (2018)], but as we have shown here, their consistency appears to be largely misunderstood as well.
In light of our findings, we recommend the following: First, published review literature on sMRI studies of PCL-R and PCL:SV psychopathy should be approached with caution, especially when the literature is used to influence forensic decisions. Second, we recommend that guidelines for conducting review literature be revised to include explicit guidance for avoiding coding bias. Although the problem of un- and under-reported null-findings is recognized [e.g., Pocock et al., 1987; Hutton and Williamson, 2000; guidelines for accurate reporting in review literature also exist; see Petticrew and Roberts (2008), American Psychological Association (2008), and Moher et al. (2015)], the role of coding bias, by and large, is not. Third, we recommend that review literature pay careful attention to the a priori likelihood of null-findings in their data. In our example, both the PCL-R (DeMatteo et al., 2020) and neuroimaging methods (Nugent et al., 2013) have relatively low reliability. The likelihood that sMRI research on psychopathy should yield more than 91% positive findings is therefore not realistic [for more extended discussions relating to fMRI, see Vul et al. (2009) and Vul and Pashler (2017)]. Fourth, we recommend that the production of new data should be complemented by closer examination of data already published. Among the 45 reviews we evaluated, we found a single study (Plodowski et al., 2009) that comprehensively reported all nulls in the original literature. Unfortunately, it was also among the least cited reviews, suggesting that accuracy and scientific impact do not necessarily go together. Finally, we recommend that reviewers pay close attention to potential biases—such as publication and reporting bias, p-hacking, and the file drawer problem—in the original literature, and take measures to compensate for them. Currently, it appears that reviews largely magnify them instead.
Limitations
Our study has a number of important limitations. First, in order to focus on forensically relevant studies, we limited our analysis to PCL-R and PCL:SV psychopathy. We also excluded studies that reported on PCL-R Factor scores only (e.g., Bertsch et al., 2013), that did not use case-control or correlational method (Sato et al., 2011; Kolla et al., 2014), and that included youth samples. It is possible that the excluded studies were reported more accurately in review literature than those we included. Second, we excluded original and review studies not published in English. This may have introduced a selection bias of our own, as it is possible that non-English publications use different standards of reporting and reviewing than those published in English. Third, our findings may have underestimated the extent of the bias. For example, one whole-brain analysis reviewed here (Contreras-Rodríguez et al., 2015) only reported positive findings, which means that the remaining brain regions were unreported nulls. Had these unreported null-findings been included in our analysis, the true percentage of nulls in the original studies would have been greater than 64.18%. Further, we did not account for possible publication bias. Since null-findings are presumed to be less likely than null-rejections to be published, the percentage of true nulls in the field is essentially unknown, though it may be significantly higher than we estimated (review literature examined here did not report any unpublished null-findings). Finally, we excluded fMRI and other imaging methods entirely. Future research could evaluate whether coding bias is present in reviews of this literature as well.
Data Availability Statement
The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author/s.
Author Contributions
Topic conceptualization was completed by JJ, SG, and RL. The PRISMA review was conducted by RL and JJ. Effect coding was conducted by JJ, SG, and BA. Coding disagreements were resolved by RL. Data analysis completed by JJ and SG. Manuscript preparation was completed by JJ, with edits from SG and RL. All authors contributed to the article and approved the submitted version.
Funding
Funding for the open access publication was provided by the University of Toronto Mississauga.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2021.654336/full#supplementary-material
Footnotes
1. ^At this step the excluded articles were Baskin-Sommers et al. (2016), Harris et al. (2001), Hoppenbrouwers et al. (2015), Kolla et al. (2014), Kolla et al. (2017), Pera-Guardiola et al. (2016), Yang et al. (2012), Beckwith et al. (2018), Bertsch et al. (2013), Leutgeb et al. (2015), Vermeij et al. (2018), Calzada-Reyes et al. (2015), Korponay (2019), Vieira et al. (2015), and Yang (2009).
2. ^At this step the excluded articles were Nickerson (2014), Blair (2019), Brower and Price (2001), Dolan (2002), Lamsma et al. (2017), and Raine (2019).
3. ^Ermer et al. (2012) found both a reduction (using cluster size) and a null (using peak height). We are thus reporting it twice in this analysis.
4. ^Targeted/focused reviews included data only on specific brain region(s) or outcome(s) (e.g., antisocial behavior).
5. ^Though, Kiehl and Hoffman (2011) refer mostly to functional MRI data, their argument is a classic example of reverse inference. They write: “(T)hese neurological results should go a long way toward ending the debate about whether psychopathy is just too difficult to diagnose to justify inclusion in the DSM. Any lingering doubts about the clinical reliability of the Hare instruments disappear now that those instruments have been shown to be robustly predictive of a demonstrable neurological condition.” (p. 390).
References
Agnoli, F., Wicherts, J. M., Veldkamp, C. L., Albiero, P., and Cubelli, R. (2017). Questionable research practices among Italian research psychologists. PLoS ONE 12:e0172792. doi: 10.1371/journal.pone.0172792
American Psychological Association (2008). Reporting standards for research in psychology: why do we need them? what might they be? Am. Psychologist. 63, 839–851. doi: 10.1037/0003-066X.63.9.839
Anckarsäter, H. (2006). Central nervous changes in social dysfunction: autism, aggression, and psychopathy. Brain Res. Bull. 69, 259–265. doi: 10.1016/j.brainresbull.2006.01.008
Anderson, N. E., and Kiehl, K. A. (2012). The psychopath magnetized: insights from brain imaging. Trends Cogn. Sci. 16, 52–60. doi: 10.1016/j.tics.2011.11.008
Anderson, N. E., and Kiehl, K. A. (2013). “Functional neuroimaging and psychopathy,” in Handbook on Psychopathy and Law, eds K. A. Kiehl and W. P. Sinnott-Armstrong (New York, NY: Oxford University Press), 131–149.
Anderson, N. E., and Kiehl, K. A. (2014a). Psychopathy: developmental perspectives and their implications for treatment. Restorat. Neurol. Neurosci. 32, 103–117. doi: 10.3233/RNN-139001
Anderson, N. E., and Kiehl, K. A. (2014b). “Psychopathy and aggression: when paralimbic dysfunction leads to violence,” in Neuroscience of Aggression, eds K. A. Miczek and A. Meyer-Lindenberg (New York, NY: Springer-Verlag Publishing), 369–393. doi: 10.1007/7854_2013_257
Aoki, Y., Inokuchi, R., Nakao, T., and Yamasue, H. (2014). Neural bases of antisocial behavior: a voxel-based meta-analysis. Soc. Cogn. Affective Neurosci. 9, 1223–1231. doi: 10.1093/scan/nst104
Baskin-Sommers, A. R., Neumann, C. S., Cope, L. M., and Kiehl, K. A. (2016). Latent-variable modeling of brain gray-matter volume and psychopathy in incarcerated offenders. J. Abnormal Psychol. 125, 811–817. doi: 10.1037/abn0000175
Bassarath, L. (2001). Neuroimaging studies of antisocial behaviour. Can. J. Psychiatry 46, 728–732. doi: 10.1177/070674370104600805
Beckwith, T. J., Dietrich, K. N., Wright, J. P., Altaye, M., and Cecil, K. M. (2018). Reduced regional volumes associated with total psychopathy scores in an adult population with childhood lead exposure. Neurotoxicology 67, 1–26. doi: 10.1016/j.neuro.2018.04.004
Bertsch, K., Grothe, M., Prehn, K., Vohs, K., Berger, C., Hauenstein, K., et al. (2013). Brain volumes differ between diagnostic groups of violent criminal offenders. Eur. Archiv. Psychiatry Clin. Neurosci. 263, 593–606. doi: 10.1007/s00406-013-0391-6
Bes-Rastrollo, M., Schulze, M. B., Ruiz-Canela, M., and Martinez-Gonzalez, M. A. (2013). Financial conflicts of interest and reporting bias regarding the association between sugar-sweetened beverages and weight gain: a systematic review of systematic reviews. PLoS Med. 10:1578. doi: 10.1371/journal.pmed.1001578
Blair, R. J. R. (2003). Neurobiological basis of psychopathy. Br. J. Psychiatry 182, 5–7. doi: 10.1192/bjp.182.1.5
Blair, R. J. R. (2008). The amygdala and ventromedial prefrontal cortex: functional contributions and dysfunction in psychopathy. Philos. Trans. Royal Soc. B Biol. Sci. 363, 2557–2565. doi: 10.1098/rstb.2008.0027
Blair, R. J. R. (2010). Neuroimaging of psychopathy and antisocial behavior: a targeted review. Curr. Psychiatry Rep. 12, 76–82. doi: 10.1007/s11920-009-0086-x
Blair, R. J. R. (2013). Psychopathy: cognitive and neural dysfunction. Dialog. Clin. Neurosci. 15, 181–190. doi: 10.31887/DCNS.2013.15.2/rblair
Blair, R. J. R. (2019). Dysfunctional neurocognition in individuals with clinically significant psychopathic traits. Dialog. Clin. Neurosci. 21, 291–299. doi: 10.31887/DCNS.2019.21.3/rblair
Blair, R. J. R., and Zhang, R. (2020). Recent neuro-imaging findings with respect to conduct disorder, callous-unemotional traits and psychopathy. Curr. Opin. Psychiatry 33, 45–50. doi: 10.1097/YCO.0000000000000559
Boccardi, M., Bocchetta, M., Aronen, H. J., Repo-Tiihonen, E., Vaurio, O., Thompson, P. M., et al. (2013). Atypical nucleus accumbens morphology in psychopathy: another limbic piece in the puzzle. Int. J. Law Psychiatry 36, 157–167. doi: 10.1016/j.ijlp.2013.01.008
Boccardi, M., Frisoni, G. B., Hare, R. D., Cavedo, E., Najt, P., Pievani, M., et al. (2011). Cortex and amygdala morphology in psychopathy. Psychiatry Res. Neuroimaging 193, 85–92. doi: 10.1016/j.pscychresns.2010.12.013
Boccardi, M., Ganzola, R., Rossi, R., Sabattoli, F., Laakso, M. P., Repo-Tiihonen, E., et al. (2010). Abnormal hippocampal shape in offenders with psychopathy. Hum. Brain Map. 31, 438–447. doi: 10.1002/hbm.20877
Boduszek, D., and Debowska, A. (2016). Critical evaluation of psychopathy measurement (PCL-R and SRP-III/SF) and recommendations for future research. J. Criminal Justice 44, 1–12. doi: 10.1016/j.jcrimjus.2015.11.004
Brower, M. C., and Price, B. H. (2001). Neuropsychiatry of frontal lobe dysfunction in violent and criminal behaviour: a critical review. J. Neurol. Neurosurg. Psychiatry 71, 720–726. doi: 10.1136/jnnp.71.6.720
Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376. doi: 10.1038/nrn3475
Calzada-Reyes, A., Alvarez-Amador, A., Galan-Garcia, L., Valdes-Sosa, M., Melie-Garcia, L., Aleman-Gomez, Y., et al. (2015). “MRI study in psychopath and non-psychopath offenders,” in Psychopathy: Risk factors, Behavioral Symptoms and Treatment Options, ed M. Fitzgerald (Hauppauge: Nova Science Publishers), 41–59. Available online at: http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=psyc12&NEWS=N&AN=2015-12905-003 (accessed November 3, 2020).
Contreras-Rodríguez, O., Pujol, J., Batalla, I., Harrison, B. J., Soriano-Mas, C., Deus, J., et al. (2015). Functional connectivity bias in the prefrontal cortex of psychopaths. Biol. Psychiatry 78, 647–655. doi: 10.1016/j.biopsych.2014.03.007
Cope, L. M., Shane, M. S., Segall, J. M., Nyalakanti, P. K., Stevens, M. C., Pearlson, G. D., et al. (2012). Examining the effect of psychopathic traits on gray matter volume in a community substance abuse sample. Psychiatry Res. Neuroimag. 204, 91–100. doi: 10.1016/j.pscychresns.2012.10.004
Craig, M. C., Catani, M., Deeley, Q., Latham, R., Daly, E., Kanaan, R., et al. (2009). Altered connections on the road to psychopathy. Mol. Psychiatry 14, 946–953. doi: 10.1038/mp.2009.40
Crooks, D., Anderson, N. E., Widdows, M., Petseva, N., Decety, J., Pluto, C., et al. (2019). The relationship between cavum septum pellucidum and psychopathic traits in female offenders. Behav. Brain Res. 359, 967–972. doi: 10.1016/j.bbr.2018.06.011
Crooks, D., Anderson, N. E., Widdows, M., Petseva, N., Koenigs, M., Pluto, C., et al. (2018). The relationship between cavum septum pellucidum and psychopathic traits in a large forensic sample. Neuropsychologia 112, 95–104. doi: 10.1016/j.neuropsychologia.2018.03.015
David, S. P., Naudet, F., Laude, J., Radua, J., Fusar-Poli, P., Chu, I., et al. (2018). Potential reporting bias in neuroimaging studies of sex differences. Sci. Rep. 8:6082. doi: 10.1038/s41598-018-23976-1
David, S. P., Ware, J. J., Chu, I. M., Loftus, P. D., Fusar-Poli, P., Radua, J., et al. (2013). Potential reporting bias in fMRI studies of the brain. PLoS ONE 8:70104. doi: 10.1371/journal.pone.0070104
de Oliveira-Souza, R., Hare, R. D., Bramati, I. E., Garrido, G. J., Ignácio, F. A., Tovar-Moll, F., et al. (2008). Psychopathy as a disorder of the moral brain: fronto-temporo-limbic grey matter reductions demonstrated by voxel-based morphometry. Neuroimage 40, 1202–1213. doi: 10.1016/j.neuroimage.2007.12.054
De Ridder, D., Langguth, B., Plazier, M., and Menovsky, T. (2009). “Moral dysfunction: theoretical model and potential neurosurgical treatments,” in The Moral Brain, eds J. Verplaetse, J. Schrijver, S. Vanneste, and J. Braeckman (New York, NY: Springer), 155–183. doi: 10.1007/978-1-4020-6287-2_7
Debowska, A., Boduszek, D., Hyland, P., and Goodson, S. (2014). Biological correlates of psychopathy: a brief review. Mental Health Rev. J. 19, 110–123. doi: 10.1108/MHRJ-10-2013-0034
DeMatteo, D., Hart, S. D., Heilbrun, K., Boccaccini, M. T., Cunningham, M. D., Douglas, K. S., et al. (2020). Statement of concerned experts on the issue of the Hare Psychopathy Checklist-Revised in capital sentencing to assess risk for institutional violence. Psychol. Public Policy Law. 26, 133–144. doi: 10.1037/law0000223
Denno, D. W. (2015). The myth of the double-edged sword: an empirical study of neuroscience evidence in criminal cases. Boston College Law Rev. 56, 493–551. Available online at: http://ir.lawnet.fordham.edu/faculty_scholarship/548
Dolan, M. (2002). What neuroimaging tells us about psychopathic disorders. Hosp. Med. 63, 337–340. doi: 10.12968/hosp.2002.63.6.2003
Douglas, T., Pugh, J., Singh, I., Savulescu, J., and Fazel, S. (2017). Risk assessment tools in criminal justice and forensic psychiatry: the need for better data. Eur. Psychiatry J. Assoc. Eur. Psychiatr. 42, 134–137. doi: 10.1016/j.eurpsy.2016.12.009
Drucker, A. M., Patrick, F., and An-Wen, C. (2016). Research techniques made simple: assessing risk of bias in systematic reviews. J. Investig. Dermatol. 136:e109–e114. doi: 10.1016/j.jid.2016.08.021
Duyx, B., Urlings, M. J., Swaen, G. M., Bouter, L. M., and Zeegers, M. P. (2017). Scientific citations favor positive results: a systematic review and meta-analysis. J. Clin. Epidemiol. 88, 92–101. doi: 10.1016/j.jclinepi.2017.06.002
Dwan, K., Gamble, C., Williamson, P. R., and Kirkham, J. J. (2013). Systematic review of the empirical evidence of study publication bias and outcome reporting bias—an updated review. PLoS ONE 8:e66844. doi: 10.1371/journal.pone.0066844
Ermer, E., Cope, L. M., Nyalakanti, P. K., Calhoun, V. D., and Kiehl, K. A. (2012). Aberrant paralimbic gray matter in criminal psychopathy. J. Abnormal Psychol. 121, 649–658. doi: 10.1037/a0026371
Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics 90, 891–904. doi: 10.1007/s11192-011-0494-7
Ferguson, C. J., and Heene, M. (2012). A vast graveyard of undead theories: publication bias and psychological science's aversion to the null. Perspectiv. Psychol. Sci. 7, 555–561. doi: 10.1177/1745691612459059
Forstmeier, W., Wagenmakers, E. J., and Parker, T. H. (2017). Detecting and avoiding likely false-positive findings–a practical guide. Biol. Rev. 92, 1941–1968. doi: 10.1111/brv.12315
Franco, A., Malhotra, N., and Simonovits, G. (2014). Publication bias in the social sciences: unlocking the file drawer. Science 345, 1502–1505. doi: 10.1126/science.1255484
Gacono, C. B. (2016). The Clinical and Forensic Assessment of Psychopathy: A Practitioner's Guide, 2 edn. New York, NY: Routledge. doi: 10.4324/9781315764474
Gao, Y. (2018). “Neurological profiles of psychopathy: a neurodevelopmental perspective,” in Routledge International Handbook of Psychopathy and Crime, ed M. DeLisi (New York, NY: Routledge), 154–165.
Gao, Y., Glenn, A. L., Schug, R. A., Yang, Y., and Raine, A. (2009). The neurobiology of psychopathy: a neurodevelopmental perspective. Can. Psychiatric Assoc. J. 54, 813–823. doi: 10.1177/070674370905401204
Glenn, A. L., and Raine, A. (2008). The neurobiology of psychopathy. Psychiatric Clin. North America 31, 463–475. doi: 10.1016/j.psc.2008.03.004
Glenn, A. L., and Raine, A. (2009). Psychopathy and instrumental aggression: evolutionary, neurobiological, and legal perspectives. Int. J. Law Psychiatry 32, 253–258. doi: 10.1016/j.ijlp.2009.04.002
Glenn, A. L., and Raine, A. (2014). Psychopathy: An Introduction to Biological Findings and Their Implications. New York, NY: New York University Press. doi: 10.18574/nyu/9780814777053.001.0001
Glenn, A. L., Raine, A., Yaralian, P. S., and Yang, Y. (2010a). Increased volume of the striatum in psychopathic individuals. Biol. Psychiatry 67, 52–58. doi: 10.1016/j.biopsych.2009.06.018
Glenn, A. L., Yang, Y., Raine, A., and Colletti, P. (2010b). No volumetric differences in the anterior cingulate of psychopathic individuals. Psychiatry Res. 183, 140–143. doi: 10.1016/j.pscychresns.2010.05.009
Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychol. Bull. 82, 1–20. doi: 10.1037/h0076157
Gregory, S., Simmons, A., Kumari, V., Howard, M., Hodgins, S., and Blackwood, N. (2012). The antisocial brain: psychopathy matters: a structural MRI investigation of antisocial male violent offenders. Archiv. General Psychiatry 69, 962–972. doi: 10.1001/archgenpsychiatry.2012.222
Griffiths, S. Y., and Jalava, J. V. (2017). A comprehensive neuroimaging review of PCL-R defined psychopathy. Aggression Violent Behav. 36, 60–75. doi: 10.1016/j.avb.2017.07.002
Hare, R. D. (2003). The Hare Psychopathy Checklist Revised (PCL-R). Toronto, ON: Multi-Health Systems, Inc.
Hare, R. D. (2016). Psychopathy, the PCL-R, and criminal justice: some new findings and current issues. Can. Psychol. 57, 21–34. doi: 10.1037/cap0000041
Harris, G. T., Rice, M. E., and Lalumeire, M. (2001). Criminal violence: the roles of psychopathy, neurodevelopmental insults, and antisocial parenting. Special Issue Psychopathy Risk Assessment 28, 402–426. doi: 10.1177/009385480102800402
Hart, S. D., Cox, D. N., and Hare, R. D. (1995). Manual for the psychopathy checklist: Screening version (PCL: SV). Toronto, ON: Multi-Health Systems.
Herba, C. M., Hodgins, S., Blackwood, N., Kumar, V., Naudts, K. H., and Phillips, M. (2007). “The neurobiology of psychopathy: a focus on emotion processing,” in The Psychopath: Theory Research and Practice, eds H. Hervé and J. C. Yuille (London: Lawrence Erlbaum Associates), 253–283.
Hoppenbrouwers, S. S., Neumann, C. S., Lewis, J., and Johansson, P. (2015). A latent variable analysis of the Psychopathy Checklist-Revised and behavioral inhibition system/behavioral activation system factors in North American and Swedish offenders. Personal. Disord. Theory Res. Treat. 6, 251–260. doi: 10.1037/per0000115
Howner, K., Eskildsen, S. F., Fischer, H., Dierks, T., Wahlund, L. O., Jonsson, T., et al. (2012). Thinner cortex in the frontal lobes in mentally disordered offenders. Psychiatry Res. Neuroimag. 203, 126–131. doi: 10.1016/j.pscychresns.2011.12.011
Hulbert, S., and Adeli, H. (2015). Spotting psychopaths using technology. Rev. Neurosci. 26, 721–732. doi: 10.1515/revneuro-2015-0025
Hutton, J. L., and Williamson, P. R. (2000). Bias in meta-analysis due to outcome variable selection within studies. J. Royal Statist. Soc. Ser. C 49, 359–370. doi: 10.1111/1467-9876.00197
Ioannidis, J. P., Munafo, M. R., Fusar-Poli, P., Nosek, B. A., and David, S. P. (2014). Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. Trends Cogn. Sci. 18, 235–241. doi: 10.1016/j.tics.2014.02.010
Jennings, R. G., and Van Horn, J. D. (2012). Publication bias in neuroimaging research: implications for meta-analyses. Neuroinformatics 10, 67–80. doi: 10.1007/s12021-011-9125-y
Johanson, M., Vaurio, O., Tiihonen, J., and Lahteenvuo, M. (2020). A systematic literature review of neuroimaging of psychopathic traits. Front. Psychiatry 10:1027. doi: 10.3389/fpsyt.2019.01027
John, L. K., Loewenstein, G., and Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23, 524–532. doi: 10.1177/0956797611430953
Jørgensen, A. W., Hilden, J., and Gøtzsche, P. C. (2006). Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ 333, 782–787. doi: 10.1136/bmj.38973.444699.0B
Kiehl, K. A. (2006). A cognitive neuroscience perspective on psychopathy: evidence for paralimbic system dysfunction. Psychiatry Res. 142, 107–128. doi: 10.1016/j.psychres.2005.09.013
Kiehl, K. A., and Hoffman, M. B. (2011). The criminal psychopath: history, neuroscience, treatment, and economics. Jurimetr. J. Law Sci. Technol. 51, 355–397.
Koenigs, M. (2012). The role of prefrontal cortex in psychopathy. Rev. Neurosci. 23, 253–262. doi: 10.1515/revneuro-2012-0036
Koenigs, M., Baskin-Sommers, A., Zeier, J., and Newman, J. P. (2011). Investigating the neural correlates of psychopathy: a critical review. Mol. Psychiatry 16, 792–799. doi: 10.1038/mp.2010.124
Köhler, M., Haag, S., Biester, K., Brockhaus, A. C., McGauran, N., Grouven, U., et al. (2015). Information on new drugs at market entry: retrospective analysis of health technology assessment reports versus regulatory reports, journal publications, and registry reports. BMJ 350:h796. doi: 10.1136/bmj.h796
Kolla, N. J., Gregory, S., Attard, S., Blackwood, N., and Hodgins, S. (2014). Disentangling possible effects of childhood physical abuse on gray matter changes in violent offenders with psychopathy. Psychiatry Res. 221, 123–126. doi: 10.1016/j.pscychresns.2013.11.008
Kolla, N. J., Patel, R., Meyer, J. H., and Chakravarty, M. M. (2017). Association of monoamine oxidase-A genetic variants and amygdala morphology in violent offenders with antisocial personality disorder and high psychopathic traits. Sci. Rep. 7:9607. doi: 10.1038/s41598-017-08351-w
Korponay, C. (2019). Structural and functional neural correlates of impulsivity: Brain imaging studies of psychopathic criminals, non-criminals, and mindfulness practitioners. Dissertation Abstracts International: Section B: The Sciences and Engineering, 80. Available online at: http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=psyc16&NEWS=N&AN=2019-41128-007 (accessed November 3, 2020).
Korponay, C., Pujara, M., Deming, P., Philippi, C., Decety, J., Kosson, D. S., et al. (2017a). Impulsive-antisocial psychopathic traits linked to increased volume and functional connectivity within prefrontal cortex. Soc. Cogn. Affective Neurosci. 12, 1169–1178. doi: 10.1093/scan/nsx042
Korponay, C., Pujara, M., Deming, P., Philippi, C., Decety, J., Kosson, D. S., et al. (2017b). Impulsive-antisocial dimension of psychopathy linked to enlargement and abnormal functional connectivity of the striatum. Biol. Psychiatry Cogn. Neurosci. Neuroimag. 2, 149–157. doi: 10.1016/j.bpsc.2016.07.004
Kvarven, A., Strømland, E., and Johannesson, M. (2020). Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nat. Hum. Behav. 4, 423–434. doi: 10.1038/s41562-019-0787-z
Laakso, M. P., Gunning-Dixon, F., Vaurio, O., Repo-Tiihonen, E., Soininen, H., and Tiihonen, J. (2002). Prefrontal volumes in habitually violent subjects with antisocial personality disorder and type 2 alcoholism. Psychiatry Res. Neuroimag. 114, 95–102. doi: 10.1016/S0925-4927(02)00005-7
Laakso, M. P., Vaurio, O., Koivisto, E., Savolainen, L., Eronen, M., Aronen, H. J., et al. (2001). Psychopathy and the posterior hippocampus. Behav. Brain Res. 118, 187–193. doi: 10.1016/S0166-4328(00)00324-7
Lam, B. Y. H., Yang, Y., Schug, R. A., Han, C., Liu, J., and Lee, T. M. C. (2017). Psychopathy moderates the relationship between orbitofrontal and striatal alterations and violence: the investigation of individuals accused of homicide. Front. Hum. Neurosci. 11:579. doi: 10.3389/fnhum.2017.00579
Lamsma, J., Mackay, C., and Fazel, S. (2017). Structural brain correlates of interpersonal violence: systematic review and voxel-based meta-analysis of neuroimaging studies. Psychiatry Res. Neuroimag. 267, 69–73. doi: 10.1016/j.pscychresns.2017.07.006
Leutgeb, V., Leitner, M., Wabnegger, A., Klug, D., Scharmuller, W., Zussner, T., et al. (2015). Brain abnormalities in high-risk violent offenders and their association with psychopathic traits and criminal recidivism. Neuroscience 308, 194–201. doi: 10.1016/j.neuroscience.2015.09.011
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P., et al. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann. Internal Med. 151, W-65–94. doi: 10.7326/0003-4819-151-4-200908180-00136
Ling, S., and Raine, A. (2018). The neuroscience of psychopathy and forensic implications. Psychol. Crime Law 24, 296–312. doi: 10.1080/1068316X.2017.1419243
Ling, S., Umbach, R., and Raine, A. (2018). “The neural basis of psychopathy,” in Routledge International Handbook of Psychopathy and Crime, ed M. DeLisi (New York, NY: Routledge), 121–135.
Loomans, M. M., Tulen, J. H. M., and van Marle, H. J. C (2013). “The psychophysiology and neuroanatomy of antisocial behavior,” in Criminal Psychology, Volume 1, ed J. B. Helfgott (Praeger. ebook), 89–116.
Lushing, J. R., Gaudet, L. M., and Kiehl, K. A. (2016). “Brain imaging in psychopathy,” in The Clinical and Forensic Assessment of Psychopathy: A Practitioner's Guide, 2nd Edn, ed C.B. Gacono (New York, NY: Routledge/Taylor & Francis Group), 32–53. doi: 10.4324/9781315764474-3
Ly, M., Motzkin, J. C., Philippi, C. L., Kirk, G. R., Newman, J. P., Kiehl, K. A., et al. (2012). Cortical thinning in psychopathy. Am. J. Psychiatry 169, 743–749. doi: 10.1176/appi.ajp.2012.11111627
Mandrioli, D., Kearns, C. E., and Bero, L. A. (2016). Relationship between research outcomes and risk of bias, study sponsorship, and author financial conflicts of interest in reviews of the effects of artificially sweetened beverages on weight outcomes: a systematic review of reviews. PLoS ONE 11:162198. doi: 10.1371/journal.pone.0162198
McGrath, T. A., McInnes, M. D., van Es, N., Leeflang, M. M., Korevaar, D. A., and Bossuyt, P. M. (2017). Overinterpretation of research findings: evidence of “spin” in systematic reviews of diagnostic accuracy studies. Clin. Chem. 63, 1353–1362. doi: 10.1373/clinchem.2017.271544
Miskovich, T. A., Anderson, N. E., Harenski, C. L., Harenski, K. A., Baskin-Sommers, A. R., Larson, C. L., et al. (2018). Abnormal cortical gyrification in criminal psychopathy. NeuroImage Clin. 19, 876–882. doi: 10.1016/j.nicl.2018.06.007
Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., et al. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systemat. Rev. 4, 1–9. doi: 10.1186/2046-4053-4-1
Moreira, D., Azeredo, A., and Barbosa, F. (2019). Neurobiological findings of the psychopathic personality in adults: one century of history. Aggression Violent Behav. 47, 137–159. doi: 10.1016/j.avb.2019.03.005
Motzkin, J. C., Newman, J. P., Kiehl, K. A., and Koenigs, M. (2011). Reduced prefrontal connectivity in psychopathy. J. Neurosci. Off. J. Soc. Neurosci. 31, 17348–17357. doi: 10.1523/JNEUROSCI.4215-11.2011
Muller, J. L. (2010). Psychopathy-an approach to neuroscientific research in forensic psychiatry. Behav. Sci. Law 28, 129–147. doi: 10.1002/bsl.926
Müller, J. L., Sommer, M., Dohnel, K., Weber, T., Schmidt-Wilcke, T., and Hajak, G. (2008). Disturbed prefrontal and temporal brain function during emotion and cognition interaction in criminal psychopathy. Behav. Sci. Law 26, 131–150. doi: 10.1002/bsl.796
Müller, V. I., Cieslik, E. C., Laird, A. R., Fox, P. T., Radua, J., Mataix-Cols, D., et al. (2018). Ten simple rules for neuroimaging meta-analysis. Neurosci. Biobehav. Rev. 84, 151–161. doi: 10.1016/j.neubiorev.2017.11.012
Murray, L., Dotterer, H. L., Waller, R., and Hyde, L. W. (2018). “Neurogenetics approaches to understanding psychopathy,” in Routledge International Handbook of Psychopathy and Crime, ed M. DeLisi (New York, NY: Routledge), 95–120.
Nadelhoffer, T., Bibas, S., Grafton, S., Kiehl, K. A., Mansfield, A., Sinnott-Armstrong, W., et al. (2012). Neuroprediction, violence, and the law: setting the stage. Neuroethics 5, 67–99. doi: 10.1007/s12152-010-9095-z
Nelson, L. D., Simmons, J., and Simonsohn, U. (2018). Psychology's renaissance. Ann. Rev. Psychol. 69, 511–534. doi: 10.1146/annurev-psych-122216-011836
Nickerson, R. S. (1998). Confirmation bias: a ubiquitous phenomenon in many guises. Rev. General Psychol. 2, 175–220. doi: 10.1037/1089-2680.2.2.175
Nickerson, S. D. (2014). Brain abnormalities in psychopaths: a meta-analysis. North Am. J. Psychol. 16, 63–78.
Nugent, A. C., Luckenbaugh, D. A., Wood, S. E., Bogers, W., Zarate Jr, C. A., and Drevets, W. C. (2013). Automated subcortical segmentation using FIRST: test–retest reliability, interscanner reliability, and comparison to manual segmentation. Hum. Brain Map. 34, 2313–2329. doi: 10.1002/hbm.22068
Nuijten, M. B., van Assen, M. A., Augusteijn, H. E., Crompvoets, E. A., and Wicherts, J. M. (2020). Effect sizes, power, and biases in intelligence research: a meta-meta-analysis. J. Intelligence 8:36. doi: 10.3390/jintelligence8040036
Ortega-Escobar, J., Alcázar-Córcoles, M. Á., Puente-Rodríguez, L., and Peñaranda-Ramos, E. (2017). Psychopathy: legal and neuroscientific aspects. Anuario de Psicología Jurídica 27, 57–66. doi: 10.1016/j.apj.2017.01.003
Orwin, R. G., and Cordray, D. S. (1985). Effects of deficient reporting on meta-analysis: a conceptual framework and reanalysis. Psychol. Bull. 97, 134–147. doi: 10.1037/0033-2909.97.1.134
Patrick, C. J. (2014). Physiological correlates of psychopathy, antisocial personality disorder, habitual aggression, and violence. Curr. Top. Behav. Neurosci. 21, 197–227. doi: 10.1007/7854_2014_345
Pera-Guardiola, V., Contreras-Rodriguez, O., Batalla, I., Kosson, D., Menchon, J. M., Pifarre, J., et al. (2016). Brain structural correlates of emotion recognition in psychopaths. PLoS ONE 11:e0149807. doi: 10.1371/journal.pone.0149807
Petticrew, M., and Roberts, H. (2008). Systematic Reviews in the Social Sciences: A Practical Guide. Hoboken, NJ: John Wiley and Sons.
Plodowski, A., Gregory, S. L., and Blackwood, N. J. (2009). “Persistent violent offending among adult men: a critical review of neuroimaging studies,” in The Neurobiological Basis of Violence: Science and Rehabilitation, eds S. Hodgins, E. Viding, and A. Plodowski (New York, NY: Oxford University Press), 137–155.
Pocock, S. J., Hughes, M. D., and Lee, R. J. (1987). Statistical problems in the reporting of clinical trials. N. Engl. J. Med. 317, 426–432. doi: 10.1056/NEJM198708133170706
Poldrack, R. A. (2011). Inferring mental states from neuroimaging data: from reverse inference to large-scale decoding. Neuron 72, 692–697. doi: 10.1016/j.neuron.2011.11.001
Poldrack, R. A., Monahan, J., Imrey, P. B., Reyna, V., Raichle, M. E., Faigman, D., et al. (2018). Predicting violent behavior: what can neuroscience add? Trends Cogn. Sci. 22, 111–123. doi: 10.1016/j.tics.2017.11.003
Pridmore, S., Chambers, A., and McArthur, M. (2005). Neuroimaging in psychopathy. Austr. N. Zeal. J. Psychiatry 39, 856–865. doi: 10.1080/j.1440-1614.2005.01679.x
Pujara, M., and Koenigs, M. (2014). “Neuroimaging studies of psychopathy,” in PET and SPECT in Psychiatry, eds R. Dierckx, A. Otte, E. de Vries, A. van Waarde, and J. den Boer (Berlin: Springer-Verlag), 657–674. doi: 10.1007/978-3-642-40384-2_28
Pujara, M., Motzkin, J. C., Newman, J. P., Kiehl, K. A., and Koenigs, M. (2013). Neural correlates of reward and loss sensitivity in psychopathy. Soc. Cogn. Affective Neurosci. 9, 794–801. doi: 10.1093/scan/nst054
Pujol, J., Harrison, B. J., Contreras-Rodriguez, O., and Cardoner, N. (2018). The contribution of brain imaging to the understanding of psychopathy. Psychol. Med. 49, 20–31. doi: 10.1017/S0033291718002507
Raine, A. (2019). The neuromoral theory of antisocial, violent, and psychopathic behavior. Psychiatry Res. 277, 64–69. doi: 10.1016/j.psychres.2018.11.025
Raine, A., Ishikawa, S. S., Arce, E., Lencz, T., Knuth, K. H., Bihrle, S., et al. (2004). Hippocampal structural asymmetry in unsuccessful psychopaths. Biol. Psychiatry 55, 185–191. doi: 10.1016/S0006-3223(03)00727-3
Raine, A., Lee, L., Yang, Y., and Colletti, P. (2010). Neurodevelopmental marker for limbic maldevelopment in antisocial personality disorder and psychopathy. Br. J. Psychiatry 197, 186–192. doi: 10.1192/bjp.bp.110.078485
Raine, A., Lencz, T., Bihrle, S., LaCasse, L., and Colletti, P. (2000). Reduced prefrontal gray matter volume and reduced autonomic activity in antisocial personality disorder. Archiv. General Psychiatry 57, 119–127. doi: 10.1001/archpsyc.57.2.119
Raine, A., Lencz, T., Taylor, K., Hellige, J. B., Bihrle, S., Lacasse, L., et al. (2003). Corpus callosum abnormalities in psychopathic antisocial individuals. Archiv. General Psychiatry 60, 1134–1142. doi: 10.1001/archpsyc.60.11.1134
Raine, A., and Yang, Y. (2006a). Neural foundations to moral reasoning and antisocial behavior. Soc. Cogn. Affective Neurosci. 1, 203–213. doi: 10.1093/scan/nsl033
Raine, A., and Yang, Y. (2006b). “The neuroanatomical bases of psychopathy: a review of brain imaging findings,” in Handbook of Psychopathy, ed C. J. Patrick (New York, NY: The Guilford Press), 278–295.
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychol. Bull. 86, 638–641. doi: 10.1037/0033-2909.86.3.638
Santana, E. J. (2016). The brain of the psychopath: a systematic review of structural neuroimaging studies. Psychol. Neurosci. 9, 420–443. doi: 10.1037/pne0000069
Sato, J. R., de Oliveira-Souza, R., Thomaz, C. E., Basilio, R., Bramati, I. E., Amaro Jr, E., et al. (2011). Identification of psychopathic individuals using pattern classification of MRI images. Soc. Neurosci. 6, 627–639. doi: 10.1080/17470919.2011.562687
Scarpazza, C., Ferracuti, S., Miolla, A., and Sartori, G. (2018). The charm of structural neuroimaging in insanity evaluations: guidelines to avoid misinterpretation of the findings. Transl. Psychiatry 8, 1–10. doi: 10.1038/s41398-018-0274-8
Schiffer, B., Müller, B. W., Scherbaum, N., Hodgins, S., Forsting, M., Wiltfang, J., et al. (2011). Disentangling structural brain alterations associated with violent behavior from those associated with substance use disorders. Archiv. General Psychiatry 68, 1039–1049. doi: 10.1001/archgenpsychiatry.2011.61
Schiltz, K., Witzel, J., Northoff, G., Zierhut, K., Gubka, U., Fellmann, H., et al. (2007). Brain pathology in pedophilic offenders: Evidence of volume reduction in the right amygdala and related diencephalic structures. Archiv. General Psychiatry 64, 737–746. doi: 10.1001/archpsyc.64.6.737
Sethi, A., Gregory, S., Dell'Acqua, F., Thomas, E. P., Simmons, A., Murphy, D. G., et al. (2015). Emotional detachment in psychopathy: involvement of dorsal default-mode connections. Cortex 62, 11–19. doi: 10.1016/j.cortex.2014.07.018
Sethi, A., Sarkar, S., Dell'Acqua, F., Viding, E., Catani, M., Murphy, D. G., et al. (2018). Anatomy of the dorsal default-mode network in conduct disorder: association with callous-unemotional traits. Dev. Cogn. Neurosci. 30, 87–92. doi: 10.1016/j.dcn.2018.01.004
Silagy, C. A., Middleton, P., and Hopewell, S. (2002). Publishing protocols of systematic reviews: comparing what was done to what was planned. JAMA 287, 2831–2834. doi: 10.1001/jama.287.21.2831
Smith, D., Smith, R., and Misquitta, D. (2016). Neuroimaging and violence. Psychiatr. Clin. North America 39, 579–597. doi: 10.1016/j.psc.2016.07.006
State v. Brian, D. (2009). No. #05 CF 3491. In the Circuit Court of Du Page County for the Eighteenth Judicial Circuit of Illinois.
Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. J. Am. Statist. Assoc. 54, 30–34. doi: 10.1080/01621459.1959.10501497
Sterling, T. D., Rosenbaum, W. L., and Weinkam, J. J. (1995). Publication decisions revisited: the effect of the outcome of statistical tests on the decision to publish and vice versa. Am. Statistician 49, 108–112. doi: 10.1080/00031305.1995.10476125
Stratton, J., Kiehl, K. A., and Hanlon, R. E. (2015). The neurobiology of psychopathy. Psychiatric Annal. 45, 186–194. doi: 10.3928/00485713-20150401-07
Szucs, D., and Ioannidis, J. P. (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biol. 15:2000797. doi: 10.1371/journal.pbio.2000797
Tiihonen, J., Rossi, R., Laakso, M. P., Hodgins, S., Testa, C., Perez, J., et al. (2008). Brain anatomy of persistent violent offenders: more rather than less. Psychiatry Res. Neuroimag. 163, 201–212. doi: 10.1016/j.pscychresns.2007.08.012
Umbach, R., Berryessa, C. M., and Raine, A. (2015). Brain imaging research on psychopathy: implications for punishment, prediction, and treatment in youth and adults. J. Criminal Justice 43, 295–306. doi: 10.1016/j.jcrimjus.2015.04.003
Vermeij, A., Kempes, M. M., Cima, M. J., Mars, R. B., and Brazil, I. A. (2018). Affective traits of psychopathy are linked to white-matter abnormalities in impulsive male offenders. Neuropsychology 32, 735–745. doi: 10.1037/neu0000448
Vieira, J. B., Ferreira-Santos, F., Almeida, P. R., Barbosa, F., Marques-Teixeira, J., and Marsh, A. A. (2015). Psychopathic traits are associated with cortical and subcortical volume alterations in healthy individuals. Soc. Cogn. Affective Neurosci. 10, 1693–1704. doi: 10.1093/scan/nsv062
Vul, E., Harris, C., Winkielman, P., and Pashler, H. (2009). Reply to comments on “Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition.” Perspectiv. Psychol. Sci. 4, 319–324. doi: 10.1111/j.1745-6924.2009.01132.x
Vul, E., and Pashler, H. (2017). “Suspiciously high correlations in brain imaging research,” in Psychological Science Under Scrutiny: Recent Challenges and Proposed Solutions, eds S. O. Lilienfeld and I. D. Waldman (Hoboken, NJ: John Wiley and Sons), 196–220. doi: 10.1002/9781119095910.ch11
Wager, T. D., Lindquist, M. A., Nichols, T. E., Kober, H., and Van Snellenberg, J. X. (2009). Evaluating the consistency and specificity of neuroimaging data using meta-analysis. Neuroimage 45, S210–S221. doi: 10.1016/j.neuroimage.2008.10.061
Wahlund, K., and Kristiansson, M. (2009). Aggression, psychopathy and brain imaging—review and future recommendations. Int. J. Law Psychiatry 32, 266–271. doi: 10.1016/j.ijlp.2009.04.007
Weber, S., Habel, U., Amunts, K., and Schneider, F. (2008). Structural brain abnormalities in psychopaths-a review. Behav. Sci. Law 26, 7–28. doi: 10.1002/bsl.802
Wolf, R. C., Pujara, M. S., Motzkin, J. C., Newman, J. P., Kiehl, K. A., Decety, J., et al. (2015). Interpersonal traits of psychopathy linked to reduced integrity of the uncinate fasciculus. Hum. Brain Map. 36, 4202–4209. doi: 10.1002/hbm.22911
Wortman, P. M., and Bryant, F. B. (1985). School desegregation and black achievement: an integrative review. Sociol. Methods Res. 13, 289–324. doi: 10.1177/0049124185013003002
Yang, Y. (2009). Structural brain imaging in psychopathy. Dissertation Abstract. Int. Sect. B Sci. Eng. 69:5804. Available online at: http://ovidsp.ovid.com/ovidweb.cgi?T=JSandPAGE=referenceandD=psyc6andNEWS=NandAN=2009-99060-091 (accessed November 3, 2020).
Yang, Y., Glenn, A. L., and Raine, A. (2008). Brain abnormalities in antisocial individuals: implications for the law. Behav. Sci. Law 26, 65–83. doi: 10.1002/bsl.788
Yang, Y., and Raine, A. (2009). Prefrontal structural and functional brain imaging findings in antisocial, violent, and psychopathic individuals: a meta-analysis. Psychiatry Res. Neuroimag. 174, 81–88. doi: 10.1016/j.pscychresns.2009.03.012
Yang, Y., and Raine, A. (2018). “The neuroanatomical bases of psychopathy: a review of brain imaging findings,” in Handbook of Psychopathy, 2nd edn, ed C. J. Patrick (New York, NY: The Guilford Press), 380–400.
Yang, Y., Raine, A., Colletti, P., Toga, A. W., and Narr, K. L. (2009a). Abnormal temporal and prefrontal cortical gray matter thinning in psychopaths. Mol. Psychiatry 14, 561–562. doi: 10.1038/mp.2009.12
Yang, Y., Raine, A., Colletti, P., Toga, A. W., and Narr, K. L. (2010). Morphological alterations in the prefrontal cortex and the amygdala in unsuccessful psychopaths. J. Abnormal Psychol. 119, 546–554. doi: 10.1037/a0019611
Yang, Y., Raine, A., Colletti, P., Toga, A. W., and Narr, K. L. (2011). Abnormal structural correlates of response perseveration in individuals with psychopathy. J. Neuropsychiatry Clin. Neurosci. 23, 107–110. doi: 10.1176/appi.neuropsych.23.1.107
Yang, Y., Raine, A., Joshi, A. A., Joshi, S., Chang, Y.-T., Schug, R. A., et al. (2012). Frontal information flow and connectivity in psychopathy. Br. J. Psychiatry J. Mental Sci. 201, 408–409. doi: 10.1192/bjp.bp.111.107128
Yang, Y., Raine, A., Lencz, T., Bihrle, S., LaCasse, L., and Colletti, P. (2005). Volume reduction in prefrontal gray matter in unsuccessful criminal psychopaths. Biol. Psychiatry 57, 1103–1108. doi: 10.1016/j.biopsych.2005.01.021
Yang, Y., Raine, A., Narr, K. L., Colletti, P., and Toga, A. W. (2009b). Localization of deformations within the amygdala in individuals with psychopathy. Archiv. General Psychiatry 66, 986–994. doi: 10.1001/archgenpsychiatry.2009.110
Keywords: psychopath, PCL-R, sMRI, review studies, systematic review, coding bias
Citation: Jalava J, Griffiths S, Larsen RR and Alcott BE (2021) Is the Psychopathic Brain an Artifact of Coding Bias? A Systematic Review. Front. Psychol. 12:654336. doi: 10.3389/fpsyg.2021.654336
Received: 15 January 2021; Accepted: 10 March 2021;
Published: 12 April 2021.
Edited by:
Agata Debowska, The University of Sheffield, United KingdomReviewed by:
Matt DeLisi, Iowa State University, United StatesDaniel Boduszek, University of Huddersfield, United Kingdom
Copyright © 2021 Jalava, Griffiths, Larsen and Alcott. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Jarkko Jalava, jjalava@okanagan.bc.ca