Dr Paul Coxon reflects on a recent Royal Society paper discussing poor scientific practices.
For better or worse, the doctrine that pervades the culture of modern academia is ‘publish or perish’. It’s the motto on everyone’s lips from postdoc to professor. Researchers at universities are graded, ranked and metricised like never before according to the number of ‘research outputs’ they can generate. Long gone are the misty days when universities were home to cloistered dons, where careers were made on a single and often largely forgotten tome. In today’s hyper-competitive academy, researchers are thoroughly drilled in the importance of creating new knowledge, and in the critical need to disseminate fresh findings by various modes – through journal articles, presenting at conferences and books. This is how science functions, and its ideas are best tested by sharing results through peer-reviewed channels with robust data from members of the scientific community.
However, institutional and funding pressures to publish has given rise to a system where careers can rise or fall according to the number of papers, which journals they were published in, and how many citations you have (I’ve checked my h-index twice today already). Concerns have been raised in a new paper by Paul Smaldino and Richard McElreath that incentivising scientists to write frequent publications with ever more novel and surprising results leads to the propagation of poor experimental practices and selective interpretation (‘cherry-picking’) of data, leading to conclusions that are likely to be incorrect.
The paper argues that ‘laboratory methods can propagate either directly, through the production of graduate students who go on to start their own labs, or indirectly, through prestige-biased adoption by researchers in other labs. Methods which are associated with greater success in academic careers will, other things being equal, tend to spread.’ Unlike large-scale industries, which rely on common standards and protocols throughout their domain, lab-based science is usually organised at much smaller scales, often based on master-apprentice-style training, which makes it difficult to implement a common set of best practices across a field. This is further hampered by the prevalence of short-term projects and high mobility of researchers between labs, countries and disciplines.
The paper goes on to say, ‘In fields such as psychology, neuroscience and medicine, practices that increase false discoveries remain not only common, but normative’. To a casual observer this may ring true, with many high-profile paper retractions from researchers working in supremely competitive medical/life sciences and allied fields. But do these issues apply in materials science?
Over coffee this week, I asked a few colleagues this exact question and, as expected, there was a range of answers. Academic materials scientists are no special breed and are under the same pressure as all scientists to publish cutting-edge research. However, the very close links between materials science and industry can offer a guard against the propagation of poor experimental practices. Many materials PhD students and postdocs take part in industrial placements — a great learning experience — and so are trained up to ‘industry standard’ early in their careers.
We agreed there should be other ways to support and reward research without the constant demand to publish. There should be an emphasis on quality, not quantity, and we should embrace new forms of scholarly communication. Emphasis should be placed on the entire process, not just the end product. The paper’s authors make clear that ‘merit takes time to manifest, and scrutinising the quality of another’s work takes time from already busy schedules’. By encouraging a long-term view of research, the drive for novelty and the temptation to cherry-pick results may be quelled and the number of reported false positives that bedevil the literature might fall. Finding a new way to assess researchers and our research will not be easy, but perhaps not impossible.
To read the paper, The natural selection of bad science, visit bit.ly/2d6Iaeh
Dr Paul Coxon is a Postdoctoral Research Associate at the University of Cambridge, UK, Department of Materials Science and Metallurgy.