100% Effective: the unrepeated studies

A few weeks back, Ben Goldacre wrote about the reproduction crisis that science is suffering from. As a very descriptive example, he addresses the large-scale use of deworming medication in developing countries which is based on a single, but very extensive study from 2004.

When Godacre described the outcome of a re-evaluation of the data from 2004, which was done in 2013, he listed all the problems found in that study – starting from missing data to wrong instructions provided by the analysis software package that was used back then. It is really no surprise that the new evaluation came to very different results about the effectiveness of deworming medication in schools.

I really appreciate that Goldacre does not take the credit from the authors of the 2004 study, acknowledging that they did a difficult and hard work in all conscience. Instead, he points out how unusual it was that those scientists provided all their raw data for a re-evaluation. And this is indeed astonishing. Goldacres comparison with the probe passing Pluto is well chosen:

Conducting a trial, and then refusing to let anyone see the data, is like claiming you’ve flown a spaceship to Pluto, but refusing to let anyone see the photos.

As a matter of fact, this happens frequently in science. As a chemist, I sometime roll my eyes when I see hundreds of numbers in the supplementary information of a paper, describing every atom coordinate obtained from a crystal structure of a molecule. But at least this tells me that I really get all the data.

When medicine is based on a single study, its effect might have been occured by chance.
Credit: BloodyMary / pixelio.de

The other and even larger problem is indeed the reproducibilty. To be sure that a results is real and well-founded, it actually needs confirmation from different scientists. It is not unusual that scientists find a protocol published, and try to build their work on that. When I go through an interesting paper, I find myself looking for loopholes of missing information that might prevent me from reproducing the result on the first place. When I do a published synthesis and I succeed on the first try, I am surprised. On the other hand, a failure might mean that I am either not skilled enough, or that some piece of information is missing in that paper.

Not to give away all the information can be essential for a scientist under the increasing pressure to “publish or perish”. Since it delays others in reproducing the work, it ensures that the scientist keeps an advantage. Authors have to fear that their manuscripts are rejected, because of a peer-reviewer who reproduces that work in his own lab, and then publishs it first.

So, hoarding data is used as an insurance of the authors, or let’s say as a “copy protection”. As understandle as this might be, this is desastrous for science, as Goldacre clearly emphasizes. Irreproducible science is basically worthless, and in the worst case harmful. I agree that this has no influence on the fact that treatment of children against worms is an urgent and important issue. But it undermines the reliability of science in our society and promotes pseudo-scientific or religious beliefs that claim to be equally justified.

Tim Hunt said the wrong thing at the wrong time

The recent incident about Tim Hunt’s comment on women in science and the viral response of female scientists all over the world was covered by the media in an extent, that was, to my opinion, quite unusual. I think, most surprised by that enormous feedback was Tim Hunt himself, who resigned from his honorary professorship and now claims to have been “hung out and dry”.

Tim Hunt. source: The Daily Beast.

I also wish to make clear that his comment is not to be justified. Especially the part about crying remembered me of my studies, when it was not uncommon that people did cry after some oral exams which were frequently held over a lab course. Studying was tough. However, the stress put onto the students was hopefully irrespective of the gender, and I think that females and males most often simply have different strategies to deal with that. Even so, it clearly states that something with the critics might be wrong when it causes such emotional reactions. In his comment, Hunt simply assumed that being a male labworker is “normal”, while being a female labworker is a deviation. This is an in insult to all people working in the lab – female and also male.

Let me tell you about my trouble with girls … three things happen when they are in the lab … You fall in love with them, they fall in love with you and when you criticise them, they cry.

My impression is, that, in addition, Hunt’s comment on women in science was simply the last straw that broke the camel’s back: Females have to face discrimination since the very beginning of science. And since recently, sexism in science was already in focus of the scientific community and also in the public. I do think that this intense anger and dispute is not harming academia. I think, it is quite healthy. Females are showing their contribution to science and call this kind of harassment what it is: patriarchalic and unfair.

One thing that seems to be not considered in the one thing is that this issue is not only concerning the lab: It applies to all fields of work, where women are a minority. There are a lot.

#distractlinglysexy. source: Mashable.com

His own defensive reaction after being that harshly treated and criticized seems a bit ironic in that context. Yes, he did something wrong. Yes, he was maybe tired and did not think before speaking. Yes, he was also maybe speaking about his personal experience. And yes, he may have even apologized. But this does not mean that everything is alright now. It seems like Tim Hunt is now the scapegoat of an angry, female Twitter mob, just because he spoke his mind. But speaking your mind does not mean that you cannot be an asshole. The same goes for having a Nobel Prize.

Hunt is 72 years old, having/being resigned from his honorary professorship. I think that this might be not a too great loss for him. I give him, that his feelings might be severely hurt. But so are many others’ feelings, too.

A lesson in academic gender bias

Last week, the Times Higher Education reported about a a paper rejection due to the fact that the two authors are female. Thankfully, this has ignited an outrage against the affected journal PLOS One, which in consequence ousted the anonymous reviewer.

source: @FionaIngleby

The rejected study focuses on gender bias in academia and concludes that there is indeed a gender bias. In this context, the rejection gives proof to that in a stunning way. But besides joining the outrage, I would like to add my opinion, since gender bias (i.e., patriarchy) is a highly sensitive and complex topic.

As a matter of fact, the majority of scientists is male and it was not too long ago that women were not accepted to be scientists at all. Also, I hardly believe that the publication would have been rejected with the same argument when all authors would have been male. The reviewer’s phrasing does not seem to imply a gender-balanced author team. Instead, it seems to aim on a contribution of supposedly missing male opinions, which is a big difference. This can only mean that the reviewer assumes that male researchers are more objective than female ones and that the female interpretation is more prone to “ideologically biased assumptions” than the male one.

The other problem is that from a man’s view, patronizing is not an issue, since we are not getting patronized. But this is also a misinterpretation, since we (men) also are affected by a gender bias which expects males give their work first priority. I think, the cases of male scientists taking one or two years off to take care for their family are rarely seen. And why? – Because it would kill our career, which is exactly what is expected from women.

Last but not least, this event also demonstrates the power of social media in science. This discussion started with a tweet, exposing the biased devaluation and insult that many authors have to face from anonymous peer review. However, double-anonymous peer review might not always be an answer – many research areas are so small that it is easy to guess who might be the author, or the reviewer, respectively.

Judging from the impact of that story, it might be worth to start a project similar to the highly acknowledged Retraction Watch. Maybe something like a “Rejection Watch”, where biased and unfair reviewer comments can be dicussed openly.

Confessional science**

In the beginning of this month, a blog hosted by the Frankfurter Allgemeine Zeitung* reported about a scientific april fool hoax: On the pre-publishing platform arXiv.org, the author Ali Frolop published what was called “A Farewell to Falsifiability“, where one of the main criteria for scientific theories is questioned. The publication date and the author name (“Ali Frolop” = “April Fool”) make clear that this is actually a joke. However, having read that quite amusing text, it appeared to me more than a seriously meant satire, than just a cheap fooling joke.

Astrology is falsifiable, and there is nothing magic about this demarcation criteria.

The concept of Falsifiability is, in my opinion, on of the most important aspects that separates science from religion. In brief, it means that a theory must allow a prove that can contradict it. Is the sun always going down every day? (Living in Iceland with the summer coming closer, I would argue about that.) The Frolop paper gives also a good overview about other criteria for scientific theories, of course putting them into question: repeatability, simplicity and a testable correctness.

So what is going on in science? The trigger for this ongoing discussion is the string theory and the resulting multiverse theory. While the string theory can be used to explain observations, it can neither be falsified, nor predict observations, which are major disadvantages for a good theory. The same is true for universes other than ours, which also are not observable yet and allow any explanation to describe a maybe not-yet-known reality.

Multiple universes. Credit: Richard Dawkins Foundation for Reason and Science.

I agree with George Ellis and Joe Silk, who rise serious concerns about the reputation of science, when the criteria for theories are weakened. For example, the theory that there is a god, is also not falsifiable, nor is it sufficient to make predictions. Which is exactly is the purpose of religion. One danger that arises from mixing these aspects of science and religion is already there: Very often, defenders of a creationist god refer to the evolutional theory as just a theory. This is absolutely correct, since evolution is testable, repeatable and simple. The hypothesis that a concious super-powerful being willingly created and altered life, is neither of them. In so far, the Frolop paper might be less a hoax than a serious concern.

This battle for the heart and soul of physics is opening up at a time when scientific results – in topics from climate change to the theory of evolution – are being questioned by some politicians and religious fundamentalists.

---

* I have to excuse myself for referring so frequently to the german media. I also follow international news, but my native language is closer to me.

** My acknowlegdment goes to Philipp Scharf, who showed me the article in the Planckton blog.

A publication bias workshop

Two weeks ago, the National Centre for the Replacement Refinement & Reduction of Animals in Research (NC3Rs) was hosting a workshop about publication bias. In that workshop, an effort was taken to bring together “funders, journals, and scientists from academia and industry to discuss the impact of publication bias in animal research”.

By this event, three very good blog articles were written from cogsci.nl , F1000 Research and one from BioMed Central. Also, a Twitter discussion was ongoing, for which I made a Storify (please feel free to give me a note if I missed something).
Judging from the distance, this workshop seemed to have had a good impact on raising the awareness about biased publication and the consequences. Also, some solutions were discussed, like prospective registration of clinical trials to journals, and new ways for publishing. To me, prospective registration might be an interesting solution, also for other disciplines. This is what every day happens when a researcher applies for funding. However, in that case, the scientist is responsible to provide all his results to the funder, but not to a journal. I agree that this idea might be complicated to manage, but I really think it is worth the effort.

Considering new ways of publishing, PLOS One seems to be a step ahead by launching a new collection focusing on negative results. As promising as this might sound in the first moment, the collection includes papers from 2008 til 2014, being published as a new collection two weeks ago, at the 25th February 2015. This still reminds me a bit of all the negative journals that are only sporadically published. Nonetheless, I think that the awareness about that issue is rising.

Discussion? Unwanted.

The dealings with research  results within the scientific community seems to become an ongoing topic in the german newspaper Die Zeit. In their online version, a case about a psychologic study is reported that might be described with words like “concealment“, “withholding“ or “suppression“. Here is what happened, as described in the article:

Frieder Lang from the University Erlangen-Nürnberg reported in March 2013 his research (please mind the paywall), which might be summarized with the insight that pessimistic people have a longer life than optimistic people. This is discussed with that pessimistic people apparently are more concerned about their health. This study, however, is questioned in some aspects by the statisticians Björn und Sören Christensen from the University of Applied Sciences and the University of Kiel. Their main concern addresses the assignment of test subjects into optimistic and pessimistic individuals. As they argue, the characteristic of an individual might change over the course of the five-years study. In my opinion, this concern does not contradict the research, but in fact gives a significant contribution.

The Zeit article reports that the Christensen brothers sent a note with their analysis in May 2013 to the journal Psychology and Aging in which the original research was published. Their note was refused for publishing, because it did not include a theoretical background between the test subjects’ felt and actual healthiness. One might argue here that delivering this background is covered by the originally addressed paper from Lang and coworkers. After this fruitless attempt to publish their concerns, that note was submitted as a paper to the Zeitschrift für Gesundheitspsychologie (Journal of Health Psychology). More than half a year later, they received again a refusal with an astonishing explanation: It was not possible to find a single referee willing to review that paper. After waiting for another half year, the manuscript was withdrawn by the authors.

In the public, science is often seen as a process of learning and exchange. Theories and conclusions can be discussed, complemented, or even overthrown. Nevertheless, the peer-review publishing system appears to be more static than a fluent research process might require. Tools like PubPeer and the review option in ResearchGate do exist and are of growing importance. But, nonetheless, they are still watched suspiciously by the established journals.

As Sören Christensen explained to me, their paper is currently under revision. So the case is not closed yet.