Research findings that are probably wrong cited far more than robust ones, study finds

Academics suspect papers with grabby conclusions are waved through more easily by reviewers

Scientific research findings that are probably wrong gain far more attention than robust results, according to academics who suspect that the bar for publication may be lower for papers with grabbier conclusions.

Studies in top science, psychology and economics journals that fail to hold up when others repeat them are cited, on average, more than 100 times as often in follow-up papers than work that stands the test of time.

The finding – which is itself not exempt from the need for scrutiny – has led the authors to suspect that more interesting papers are waved through more easily by reviewers and journal editors and, once published, attract more attention.

“It could be wasting time and resources,” said Dr Marta Serra-Garcia, who studies behavioural and experimental economics at the University of California in San Diego. “But we can’t conclude that something is true or not based on one study and one replication.” What is needed, she said, is a simple way to check how often studies have been repeated, and whether or not the original findings are confirmed.

The study in Science Advances is the latest to highlight the “replication crisis” where results, mostly in social science and medicine, fail to hold up when other researchers try to repeat experiments. Following an influential paper in 2005 titled Why most published research findings are false, three major projects have found replication rates as low as 39% in psychology journals, 61% in economics journals, and 62% in social science studies published in the Nature and Science, two of the most prestigious journals in the world.

Working with Uri Gneezy, a professor of behavioural economics at UCSD, Serra-Garcia analysed how often studies in the three major replication projects were cited in later research papers. Studies that failed replication accrued, on average, 153 more citations in the period examined than those whose results held up. For the social science studies published in Science and Nature, those that failed replication typically gained 300 more citations than those that held up. Only 12% of the citations acknowledged that replication projects had failed to confirm the relevant findings.

The academic system incentivises journals and researchers to publish exciting findings, and citations are taken into account for promotion and tenure. But history suggests that the more dramatic the results, the more likely they are to be wrong. Dr Serra-Garcia said publishing the name of the overseeing editor on journal papers might help to improve the situation.

Prof Gary King, a political scientist at Harvard University, said the latest findings may be good news. He wants researchers to focus their efforts on claims that are subject to disagreement, so that they can gather more data and figure out the truth. “In some ways, then, we should regard the results of this interesting article as great news for the health of the scholarly community,” he said.

Prof Brian Nosek at the University of Virginia, who runs the Open Science Collaboration to assess reproducibility in psychology research, urged caution. “We presume that science is self-correcting. By that we mean that errors will happen regularly, but science roots out and removes those errors in the ongoing dialogue among scientists conducting, reporting, and citing each others research. If more replicable findings are less likely to be cited, it could suggest that science isn’t just failing to self-correct; it might be going in the wrong direction.’

“The evidence is not sufficient to draw such a conclusion, but it should get our attention and inspire us to look more closely at how the social systems of science foster self-correction and how they can be improved,” he added.


Ian Sample Science editor

The GuardianTramp

Related Content

Article image
Wellbeing enhanced more by places than objects, study finds
Research using brain scans finds people experience feelings of contentment from places more than from objects such as photographs or wedding rings

Caroline Davies

12, Oct, 2017 @2:30 PM

Article image
Why is so much research dodgy? Blame the Research Excellence Framework
The Ref star system encourages novelty but offers no incentive to replicate studies – and that’s exactly what scientists need to do to be more sure of our claims

Alex Jones and Andrew Kemp

17, Oct, 2016 @9:07 AM

Article image
Drugs and alcohol do not make you more creative, research finds
Travel, meditation, training and exposure to culture have a greater effect on artistic output

Rachel Hall

24, Mar, 2023 @3:00 PM

Article image
Universities benefit from recessions as they recruit more talented researchers

Research shows that economics PhD graduates in the US who applied during recession times had better publication rates

Michael Boehm and Martin Watzinger

07, Mar, 2014 @1:59 PM

Article image
Mental illness link to art and sex

From Lord Byron to Dylan Thomas and beyond, the famous philanderers of the art world may have had a touch of mental illness to thank for their behaviour, psychologists report today.

Ian Sample, science correspondent

30, Nov, 2005 @10:29 AM

Article image
Breast cancer study finds 'astonishing' drug combination that gives results
Using Herceptin in combination with another drug can shrink tumours in less than two weeks, study finds

Haroon Siddique and Kevin Rawlinson

11, Mar, 2016 @7:08 AM

Article image
Face masks make people look more attractive, study finds
Images of men wearing a blue medical face mask perceived as being the most attractive

Steven Morris

13, Jan, 2022 @5:12 PM

Article image
Oxford and UCL tipped to win lion’s share of grants in UK research audit
Northumbria soars up research excellence framework ratings of 76,000 academics used to divide up £2bn annual funding

Richard Adams, Education editor

11, May, 2022 @11:01 PM

Article image
Big city indifference to strangers may be a myth, study suggests
Behavioural experts in London find socio-economic factors to be the keys to helpfulness

Natalie Grover Science correspondent

07, Oct, 2020 @5:00 AM

How to become more attractive

The secret of attractiveness is making other people smile, according to a study by British psychologists. Experiments at Aberdeen University found we are more likely to think a person is good-looking if we catch members of the opposite sex smiling at them.

Ian Sample, science correspondent

17, Jan, 2007 @3:19 PM