The scientists who claimed to have observed a neutrino travelling faster than light said it could change the way we understand our universe, and yet nobody had independently checked their research. I’m a trained physicist, and along with every other physicist I’ve spoken to, I thought that a faster than light neutrino was a ridiculous result.
The fact that the results had not been checked by anyone outside the research group that published the paper concerned me. When you are close to a study, you sometimes miss the obvious. The story nevertheless made headlines around the world in September.
A month later the Berkeley Earth Surface Temperature (BEST) project released a set of temperature data showing that since the 1950’s, the Earth’s land temperature had warmed 1°C. “Global warming is real,” boomed the press release. NASA, the National Oceanic and Atmospheric Administration, and the Met Office have each independently reached this conclusion, but it was BEST’s results that the media lapped up.
BEST’s decision to release its findings caused ructions in the online scientific community, not for their conclusions, but because they had not been peer reviewed. The lead researcher at BEST, Richard Muller, is known for being outspoken. His brusque New York accent enhances this impression, but he is very highly regarded in academic circles. So why did he not wait until his papers had been reviewed before releasing them to the media? What gave him the right to subvert the usual process?
Muller is blowing a raspberry in the face of 350 years of scientific structures. The first pre-publication peer review happened in 1665, for the Philosophical Transactions of the Royal Society. Peer review became standard for many fields during the 20th century. Physical Review Letters, a quick-publication high-energy physics journal, had a consistent peer review process from its inception in the 1950s. Some fields, such as medicine, lagged behind – by 1985, The Lancet, one of the most respected journals in the field, sent only 30 per cent of papers for outside review.
But the current system is “a tyranny [large journals] have been trying to enforce”, Muller tells me. In 1969, the editor of the New England Journal of Medicine, Franz Ingelfinger, implemented the rule that work published in his journal must not have previously been publicly available anywhere. Today, several journals have this rule, preventing scientists from showcasing their own work elsewhere.
Nowadays, once a paper has been sent to a journal, the journal sends it to three or four experts in the field. These anonymous peers comment on the paper and sometimes suggest improvements. The paper is then returned to the researcher for any changes, or published if judged good enough by the reviewers and the journal’s editor. Once accepted, if the journal or the researchers think that the discovery might fire the imagination of the general public, they will distribute press releases in the hope of winning coverage.
BEST never needed much of a media machine, because it was always interesting to journalists. The initiative was established in the wake of the “climategate” scandal, and funded by billionaire oil barons the Koch brothers. In March, BEST released preliminary results of trends in global surface temperature. The data, analysis and four papers are available on the project website.
Preventing mistakes
But those four papers have not yet been peer reviewed. Muller says that his primary reason for releasing the results to a media fanfare was to stop journalists making mistakes. Concurrently to publishing his data, he wrote an op-ed in the Wall Street Journal, “largely to keep newspapers from misinterpreting the results”. It didn’t work – Muller reckons that two-thirds got “the whole sense of what we did wrong”.
Bob Ward, policy and communications director of the Grantham Research Institute on Climate Change and the Environment, thinks pre-empting the standard checks is “a dangerous strategy…[because] there’s always the possibility that they’re wrong”. Peer review emerged for a reason: to provide a systematic way of judging work.
Climate science has been in trouble for errors before. In 2009 a sharp-eyed British geographer, Professor Graham Cogley, spotted an error in a mammoth Intergovernmental Panel on Climate Change (IPCC) report, and climate change deniers leapt on the figure. Around the same time, the Climategate scandal broke. The media started to question climate science and its methods. Global warming deniers were roused and started spouting spurious claims in mainstream media of errors in the science. If there were an error in BEST, “[climate change] sceptics would have a field day”, says Ward.
Muller says that with BEST he went through what he terms “traditional” peer review. “First you peer review in your own group,” he tells me. “Then you show it to people at Berkeley, friends and so on. Then you send it out across the country and across the world to a few selected people and then you make it available to everybody… None of this ‘let’s keep it hidden until it’s published in Science and Nature’.”
Dozens of scientific papers are published before peer review every day, although usually without any media hullabaloo. Almost every physics or maths paper is deposited online at the arXiv (pronounced archive) and is critiqued before being submitted to a journal. But in general, the public care much more about climate change than condensed matter physics, and so the media keep their noses out of the arXiv. BEST is looking at global warming, an issue that directly affects the lives of everyone in the world, so the media tail its progress much more carefully.
Ivan Oransky, who runs the blog Retraction Watch, agrees with Muller’s sentiment. “We need to stop treating papers like they are a sacred relic,” he says. Science doesn’t begin or end with the production of a paper, argues Oransky, who believes that journalists need to be better at looking at the broader picture.
At the moment they don’t seem to be doing a very good job. Journalists love to write exclusives on unpublished research, but a 2002 study showed that this strategy can be problematic. According to the research, papers presented at conferences are often reported by the media, but only 53 per cent of those that make it into newspapers are ever published in important scientific journals. It made no difference if the conference provided a press release to accompany the research; the publication rate remained the same.
Famous, but not rigorous
Muller believes that his checks eliminate serious errors, and doubts that a journal’s peer reviewers would have a high chance of finding any that existed. He clearly has a problem with Nature and Science, the two most influential academic journals. “Everybody who’s submitted articles to journals like this has gotten anonymous peer reviews that are outrageous,” he says.
He recalls one paper he submitted in the 90s on paleoclimate (the study of ancient weather data). Rather than using the statistical analysis method other paleoclimatologists used, he asked the “top statisticians” in the world for the best way to analyse his data. They recommended the Lomb method, a known tool for dealing with unevenly spaced data. “[The paper] was rejected by a referee for using some obscure statistical method instead of the usual one that everyone else uses,” says Muller. “Anybody who does something innovative has a much harder time getting published.” Muller has submitted the BEST papers for peer review, but not to Nature or Science.
But there is another reason for Muller to court media attention. Climate scientist and commenter Judith Curry claims that “The publicity is so that the IPCC can’t ignore BEST”. The IPCC is in the process of writing its next report and requires submissions by November, so Muller could not hang around waiting to release his results.
The irony of the debate is that, unlike the speeding neutrinos, the science behind BEST is uncontroversial. Muller says he was doing what other climate scientists had “always wanted to do” but couldn’t because when they set up their experiments decades ago, they lacked computing power. Nick Jelley, a professor of physics at Oxford University, says: “It’s just a verification, not really anything original.” Nor, judging by the online reaction, have climate change deniers been convinced. “They’ve all trashed this work because it didn’t come up with the result they wanted,” says Ward.
But the debate is still vital. Over a million papers were published in 2009; no individual can judge each of these papers on their own. Peer review sorts the wheat from the chaff. It filters important scientific work into the most famous journals, and the process often improves papers. If it becomes standard to bypass the system, the media becomes the filter – and journalists are not scientific experts. Oransky paraphrases Churchill – it’s the “absolute worst” system we have, but there is nothing better.