In September this year, David Crotty wrote an blog post about two colliding crises – each in the context with negative results. The first crisis is described as a “reproducibility crisis, based on the assumption that a majority of published experiments is in fact unreproducible. The second crisis is referred to as “negative results crisis”, describing that a large amount of correct results remains unpublished, due to its null-result character. Both crises are described to cause a considerable waste of time for scientists – either in performing published experiments that however cannot succeed, or by repeating unsuccessful experiments that have not been published.
One attempt to overcome the problem of negative results was suggested by Annie Franco, Neil Malhotra and Gabor Simonovits, namely by “creating high-status publication outlets for these studies”. Bu I have to agree that this is easily said.
How willing are researchers to publicly display their failures? How much career credit should be granted for doing experiments that didn’t work?
Even though theses problems are clearly not new (I decidated this blog to negative results for a reason), I was surprised to see them actually described as “crises”. I do think that there is a problem of science losing trust by the public, caused by the omnipresent publish-or-perish paradigm.
Are authors of a scientific publication really more unwilling to share their raw data when their reported evidence is not too strong? This question was recently addressed in the subject of psychology, unsurprisingly published in the open-access journal PLoS ONE. Jelte Wicherts, Marjan Bakker and Dylan Molenaar from the Psychology Department of the University of Amsterdam, indeed came to that conclusion. Their study included 1149 results from 49 papers. It is interesting that in 28 of the considered 49 papers, the respective co-authors did not share their research data, even if they had agreed on that before.
Distribution of reporting errors per paper for papers from which data were shared and from which no data were shared. From DOI 10.1371/journal.pone.0026828
However, one might argue that the authors of this interesting “meta”-study walk on a difficult terrain, as they are trying to draw a correlation about the accuracy of other scientists’ correlations. But I think, their paper makes it clear enough that they were very much aware of that issue.
This week, on Wednesday, a scientific tweetstorm started about a publication from Zachary W. Culumber et al., who published a paper without removing a draft comment that was not supposed to get public. The full story is nicely summarized by Grrlscientist.
Despite the fact that many of the commenters address the apparent lack of proper peer-reviewing before the manuscript was published, I had another thought. What if…?
Typically, a manuscript is reviewed thoroughly by the authors themselves before it gets submitted. Then, in the following peer-review process, at least two anonymous experts are reviewing the text and offering comments, including their suggestion if this manuscript should be either accepted or rejected for publication. It is unlikely, but of course not impossible, that such a blatant mistake got overlooked. And, the peer-reviewers are not paid and doing that work besides their actual one (like teaching classes, supervising research, or applying for funding). But what if… this was done on purpose?
My theory is that the reviewers, or at least one of them, might be a competitor of the authors. It might have happened that this mistake was noticed and not commented. In that case, the anonymous peer-review process would offer a perfidious way to harm competitors – simply by letting them run into the open knife.
However, in each case the peer review had clearly failed. It’s only human and besides a gleefully smiling community, the scientific results were never called into question.
When my attempt to write about negative results in this blog was told in a small discussion, a friend mentioned that there already is a journal covering “null” results in science. So, I would like to address the “Journal of Unsolved Questions” (JUnQ).
Since I was unaware of this journal, I was accordingly surprised that the journal is very alive with (as far as I can judge) two issues per year, being published by PhD students from the university of Mainz, Germany. The journal features articles, guest contributions, and comments from contributors around the world, covering various scientific topics. The articles are peer-reviewed and judged for acceptance or refusal by independent referees. Also, it seems very consistent with the journal’s name that most of the articles’ titles are indeed questions, which is refreshing since scientists are usually supposed to offer answers instead. Personally, I took a great interest in the article of Natascha Gaster, Jorge S. Burns and Michael Gaster about the ICMJE recommendations and the problem of co-author overflow and honory authorships in articles.
Nonetheless, it occurs to me that in JUnQ – although dedicated to “[…] making ‘negative’ and ‘null’-results from all fields of science available to the scientific community” – the authors rephrase the “null” outcomes of their work to open questions. That’s fair enough, since negative results do keep the original questions unsolved, or even give rise to new ones.
What I am still wondering about is whether there is a similarly serious platform for experimental studies with a “true negative” outcome. JUnQ is clearly contributing to a manifold of unsolved questions in sciences, but I think a platform for negative experimental results would help scientists to avoid running into dead ends that had been already discovered, but never published.
It’s Nobel Prize week. And as everyone knows, the Nobel Prize is considered to be the highest award a scientist can get in his carreer. This award is so archetypical, that the secret striving to eventually get the Nobel Prize is ascribed to every one doing science, and is often used in movies and TV shows as a typical cliché.
In terms of “pure” science, the aim for reputation and ackowledgement of scientists might seem somehow disturbing. The first motivation of a scientist should not be to achieve an award or to gain reputation – it should be to solve a distinct problem, and to learn something new about nature. Of course, this image of a selfless scientist who works only in duty of finding the pure truth is as wrong as the assumption of something like “the pure truth” at all. Scientists do research because it is their job. They have studied, they have to fulfill contracts and they want to have a good and wealthy life, as everyone else does. And, of course, scientists also want to be acknowledged for their work, no less than everyone else does.
I think, the Nobel Prize is a perfect example about expectations. Getting the Nobel Prize is virtually impossible and purposefully working on getting the Nobel prize cannot be an option. The only thing one can really do is doing the best possible work and to hope that later, people notice that this work was really contributing to the progress of our society. And this is what the Nobel Prize is for.
So there are many good reasons to acknowledge the successes of the rewarded people and to do the best possible scientific work.
I recently read a text about the concept behind a scientific publication, stating that it is somewhat misleading when it comes to the description of the scientific process. True enough, most papers are built upon a theory that is supposed to be tested, followed by a respective experimental setup in order to prove that theory. Nevertheless, this is indeed not how science usually works. The most important breakthroughs are coming from sidetracks, unexpected observations, or even from miscarried experiments.
I have to agree that the process of deduction cannot produce any information that has not been there before. Accessing and combining given information is clearly an important factor in science, but I think that it is difficult to conclude previously unknown concepts, or question the established ones only by deduction. But this is, however, what the structure of most scientific articles pretends: The team of scientists has an enlightenment about a given theory and deducts a meaningful experiment to prove or disprove exactly defined aspects of that theory. The data is then collected and listed without any subjective interpretation at this point of time. Finally, when all this is done, for the first time the scientists look at their new data in context of the theory to prove and come to new, ground-breaking conclusions about nature. I would be interested, how many of those publications originate from an experiment that was supposed to give a completely different result and let the researchers being puzzled for considerable time.
In this essay, „The importance of stupidity in scientific research“, Martin Schwartz arises the conflict between the consideration of scientists as smart people, while many scientists themselves instead feel stupid in their work. Scientists are indeed addressing problems that not so many people have addressed before – which is the reason why they do it. So clearly, there is a lack of certainity, and every step has to be done carefully. It happens so easily that something gets overlooked, misinterpreted, or overrated. In science, you don’t simply know.
If you realize that you don’t know too much about certain things, and these things happen to be your scientific project, you must feel quite stupid. And again, this is why we do science: because we don’t know things.
The current retraction wave in Nature is still highly discussed in the scientific community. Indeed, as of September 2014, the number of retractions is 8, which is yet even higher than in 2013, where it have been „only“ six retractions. In this discussion, the record from 2003 is often referred to, which is supposed to be 10.
So, what is going on in Nature? Paul Knoepfler addresses this question intensively in his blog, also pointing out that the increased number of retraction might be the result of a lower tolerance by the staff of Nature. Although the numbers of retractions over the last years look impressive (1–2–6–8, ranging from 2011 to 2014), this high number of retractions looks different in comparison to the record in 2003. Nevertheless, I have to contradict @Richvn, as two of the ten listed papers are related to retractions, but are not actual retractions.
The contribution from my side to the ongoing discussion about Nature‘s wave of retractions is therefore, that this at least is not unique in the history of Nature. Nevertheless, 2014 has not passed yet. And several publications are, of course, retracted at a later point of time.