Replication has been a big topic in psychology recently as a) we've suddenly realised we need more of it and b) because there have recently been several high profile replication efforts published (e.g the Many Labs effort; see Ed Yong's summary). Last week Simone Schnall, one of the authors whose work failed to replicate in that project, wrote a blog about her experiences getting replicated. She has issues with the replication (specifically that their data had a ceiling effect on the morality measures which would obscure any differences) but one particular comment caught my eye:
Of course replications are much needed and as a field we need to make sure that our findings are reliable. But we need to keep in mind that there are human beings involved, which is what Danny Kahneman’s commentary emphasizes. Authors of the original work should be allowed to participate in the process of having their work replicated. (emphasis mine)
This idea that there is somehow a requirement to involve authors in efforts to replicate their work. This is nonsense; once you have published some work then it is fair game for replication, failure to replicate, criticism, critique and discussion. In other words, we're all allowed to science the hell out of your work any time we like. We don't need either your permission or your involvement: the only thing we (should) need is your Methods section and if you don't like this, then stop publishing your results where we can find them.
Of course, getting the original authors involved can be very productive; you can chat experimental details, make sure you have covered everything, and generally be collegial about the whole thing instead of adversarial. But there is no obligation to do this, and I'm surprised that people think there is one.
UPDATE 9 June 2015: We published this critique.
Around about the time we published our embodied cognition paper, we also reviewed a paper for that research topic, which happened to be an example of the conceptualisation hypothesis style of 'embodied' cognition we aren't all that impressed by.
We had serious reservations about the paper (detailed below) and did not think it should be published. After several rounds of trying and failing to get the authors to acknowledge the problem, we withdrew from the review and the editor, Dermot Lynott then decided to side with the other reviewers (who had identified the same problem we had but who didn't think it was a fatal flaw). The paper (Dijkstra, Eerland, Zijlmans & Post, 2012) was therefore published.
A while back I drafted a commentary laying out the problems with this paper; the full text is here. The motivation for a comment was the same as for the one I wrote for Soliman et al (2014); this style of embodiment does not tackle the hard questions about mechanism that are crucial for an embodied account and this is a big problem. I cannot decide if the commentary is worth publishing; this paper is not that important, although it is a good example of this major issue for 'grounded' embodied cognition. Any thoughts on this would be welcome.
Posts
