Researchers find that only 3% of the psychology journals explicitly state they accept these non-novel, replication studies for submission. Image by Shutterstock.

Replication crisis in science highlighted by low acceptance rate of replication studies in psychology journals. Researchers find that only 3% of the psychology journals explicitly state they accept these non-novel studies for submission.  

— By Conn Hastings

Story in Brief

It is important that scientific experiments are repeatable and produce identical or similar results if repeated. Otherwise, it is difficult to know if experimental results are robust and reveal a real phenomenon, or if they are merely a one-off caused by experimental error or highly specific conditions that are difficult to recreate.

In replication studies, scientists try to replicate the findings of previous studies to verify that their results are robust. Recently, a so-called “replication crisis” has been brewing in science. There is a growing awareness and discussion in the wider scientific community that replications are not performed or published enough. This could potentially result in whole areas of scientific research being constructed on foundations of sand.

The tendency of scientific journals to prefer to publish positive and original research contributes to the replication crisis. Researchers in London have investigated 1151 psychology journals and found that just 3% state that they welcome scientists to submit replication studies for publication and 33% emphasized the need for scientific originality in submissions, which discourages scientists to submit replications. If scientific journals become more open to publishing replications, this can help to address the replication crisis.

Researchers in London have investigated 1151 psychology journals and found that just 3% state that they welcome scientists to submit replication studies for publication. In replication studies, scientists try to replicate the findings of previous studies to verify that their results are robust. By not explicitly welcoming these studies, scientific journals might deepen the replication crisis and counteract science to move forward by these qualitative self-corrections.

There are three hurdles to overcome this replication crisis: the perception of scientists, funding bodies, and journals. Publishing research findings in scientific journals is a career yardstick for many scientists, where the number and perceived impact of their publications are considered important measures of career success. This view is also frequently taken by funding bodies and tenure committees that provide academic researchers with hotly contested research funding and job security.  And lastly, many journals tend to only accept papers that report positive and original findings: the so-called publication bias

Scientific journals provide a platform for scientists to communicate their research. The impact of published papers is often gauged by how many times they are cited (or referenced) by other papers, the idea being that such citations indicate that other scientists are building on the original work.

Scientific journals are also judged on the citations their published manuscripts receive, leading to metrics such as the impact factor. Put simply, the impact factor represents the average number of citations an article in a journal receives in a year. Journals with higher impact factors are often considered to be more desirable places to publish, although the impact factor has sometimes been criticized as an inaccurate measurement of quality.

In traditional academic publishing, a small minority of submitted studies are accepted for publication, based on their perceived significance or originality, or if they confirm an existing hypothesis. Studies that are thought to provide only a small advance, or those that present “non-impactful” or negative results are frequently rejected. In fact, many journals explicitly state in their aims or guide to authors that high levels of significance or originality are a prerequisite for publication. Journal editors often enforce extremely high rejection rates, the idea being that selective publishing will increase the journal’s Impact Factor.

However, this traditional publishing model has drawn criticism that it may exclude studies with perceived low impact, but which are valuable for scientific integrity and development. Replications are one such type of study.

“Science progresses through replication and contradiction. The former builds the body of evidence, the latter determines whether such a body exists,” explains Professor Neil Martin, of Regent’s University London, lead author on the study, recently published in Frontiers in Psychology.

The replication crisis has been brewing in science, and has recently come to a head following some high-profile failed replications in psychology. “Researchers have been accused of various creative methodological misdemeanors which may have led to false-positive results being published,” says Martin. “We’re still uncovering questionable research practices in some well-known historical studies, and I would not be surprised to see many others emerging.” Moreover, there is a growing awareness and discussion in the wider scientific community that replications are not performed or published enough. This could potentially result in whole areas of scientific research being constructed on foundations of sand.

There are many reasons for the lack of replication studies in various fields of science. The overriding scientific culture is one of innovation, originality and discovery, and scientists may be reluctant to conduct “housekeeping” replication studies, when resources are limited.

“Journals have been criticized for not readily accepting replication studies, but the basis for this criticism is anecdotal,” explains Martin.

To begin to quantify this phenomenon, Martin and Richard Clarke, a research student at the London School of Hygiene and Tropical Medicine, investigated how welcoming psychology journals are to publishing replications. “We wanted to investigate whether journals specifically rejected (or did not recommend) the submission of replications. We did this by examining the aims and instructions to authors of 1151 journals in psychology,” says Martin.

The team found that only 3% of the psychology journals on their list explicitly stated that they accepted replications. There was no difference between high and low impact journals and no difference between the different branches of psychology. 33% of the journals emphasized the need for scientific originality in submissions, which discourages scientists to submit replications, while 63% neither encouraged nor discouraged replications. The remaining 1% actively discouraged replications.

So how do we make psychology journals more welcoming to replications? “We’ve suggested that all journals in psychology should state that they accept replications that are positive and negative,” explains Martin. “Researchers could also submit two papers for publication when they submit original research: one which reports the original results and one replication which acts as a test of the original findings.”

An alternative approach to traditional publishing, in the form of impact-neutral publishers, could also help to increase the number of published replication studies. Impact-neutral publishers, such as PLOS ONE, Frontiers and many BMC journals, don’t make value judgements on perceived impact, and instead assess the scientific validity of a submitted study, when deciding whether to publish it. “If a study tests a hypothesis based on sound reasoning, using sound methodology with appropriate data analysis, those should be the absolute criteria for publishing,” says Martin.

Impact-neutral publishing may help researchers to address the replication crisis, by creating a positive atmosphere in which to publish replications. Despite not publishing articles on the basis of perceived impact and having a low overall rejection rate (~27%), the impact of Frontiers journals is significant and increasing. For example, Frontiers journals rank in the top 15% of impact factors for all journals, based on Journal Citation Reports published in 2016. The success of newer impact-neutral publishers is thus challenging the traditional belief that impact-based selectivity is the best way to publish impactful and quality research.

Some funding bodies and foundations are also acknowledging their role in making replication research a more mainstream part of scientific practice. The Netherlands Organisation for Scientific Research, for example, has launched the world’s first national fund dedicated to replication studies. And the German Research Foundation recently published a statement in which they acknowledge their responsibility in the replication crisis debate. They will develop criteria for them to fund replication studies. With resources coming more available to scientists, it is hoped that replication studies will not be seen anymore as conducting housekeeping studies: they are actually the foundation blocks of scientific integrity.