Psychology's real replication problem: our Methods sections

Replication has been a big topic in psychology recently as a) we've suddenly realised we need more of it and b) because there have recently been several high profile replication efforts published (e.g the Many Labs effort; see Ed Yong's summary). Last week Simone Schnall, one of the authors whose work failed to replicate in that project, wrote a blog about her experiences getting replicated. She has issues with the replication (specifically that their data had a ceiling effect on the morality measures which would obscure any differences) but one particular comment caught my eye:

Of course replications are much needed and as a field we need to make sure that our findings are reliable. But we need to keep in mind that there are human beings involved, which is what Danny Kahneman’s commentary emphasizes. Authors of the original work should be allowed to participate in the process of having their work replicated. (emphasis mine)

This idea that there is somehow a requirement to involve authors in efforts to replicate their work. This is nonsense; once you have published some work then it is fair game for replication, failure to replicate, criticism, critique and discussion. In other words, we're all allowed to science the hell out of your work any time we like. We don't need either your permission or your involvement: the only thing we (should) need is your Methods section and if you don't like this, then stop publishing your results where we can find them.

Of course, getting the original authors involved can be very productive; you can chat experimental details, make sure you have covered everything, and generally be collegial about the whole thing instead of adversarial. But there is no obligation to do this, and I'm surprised that people think there is one.

Connecting the conceptual dots in embodied cognition

UPDATE 9 June 2015: We published this critique. 

Around about the time we published our embodied cognition paper, we also reviewed a paper for that research topic, which happened to be an example of the conceptualisation hypothesis  style of 'embodied' cognition we aren't all that impressed by.

We had serious reservations about the paper (detailed below) and did not think it should be published. After several rounds of trying and failing to get the authors to acknowledge the problem, we withdrew from the review and the editor, Dermot Lynott then decided to side with the other reviewers (who had identified the same problem we had but who didn't think it was a fatal flaw). The paper (Dijkstra, Eerland, Zijlmans & Post, 2012) was therefore published.

A while back I drafted a commentary laying out the problems with this paper; the full text is here. The motivation for a comment was the same as for the one I wrote for Soliman et al (2014); this style of embodiment does not tackle the hard questions about mechanism that are crucial for an embodied account and this is a big problem. I cannot decide if the commentary is worth publishing; this paper is not that important, although it is a good example of this major issue for 'grounded' embodied cognition. Any thoughts on this would be welcome.

Does action scaling predict 'the embodiment of culture'? (No.)

I recently reviewed a paper for Frontiers by Arthur Glenberg and colleagues called 'Sensory motor mechanisms unify psychology: the embodiment of culture' (Soliman, Gibson & Glenberg, 2014). This is part of an ongoing research topic on embodied cognition run by Guy Dove. Once the paper had been published, I took the opportunity to write up my main remaining problem with the paper as a commentary piece (Wilson, 2014) and I'd like to review that here (and also please see my comment at the end about my role here as reviewer). Go download the paper, though, I worked hard to produce a focused critique in the 1000 word limit and I think it went well.

Glenberg and colleagues are trying to develop a broad embodied framework that can encompass both traditional hunting grounds like perception and action but also 'higher order' cognition like social cognition and culture. I admire the effort but to my mind this is grounded cognition, not embodied and so this is not an approach I'd endorse. I like to review papers like this though because it's healthy to have an adversarial reviewer who tries not to be evil (I promise!), and because I also like to keep myself informed about the work I'm rejecting so that I don't start fighting straw men.

What miming a steering wheel tells us about what we learn

Pro-tip - keep your eyes open while doing this

One of my favourite podcasts is '99% Invisible' by Roman Mars. It's about design, and about the consequences for good and ill design has on our day to day life. You should listen to it, because it is awesome. 

An older episode fell into my ball park a little bit, and I thought it was a nice idea worth repeating. It's about the steering wheel, and what we learn when we learn to use them. The moral of the story is this: when we learn, we don't acquire internal models of the features of a task which we can then access later on. Instead, we learn how to interact with a given task dynamic and how to use the information made available by that task dynamic.

Help with a lay summary of my throwing research

I am applying for a research fellowship and part of the application is a 2500 character lay summary; see below. I would love all and any feedback from you about how well it scans for you.

I have also put a public copy of this and the scientific abstract on Google Drive that can take comments, if you are so inclined:
Lay summary
Scientific abstract

The remit of the section is
Provide a lay summary of your proposed project. This should be understandable by an A-level science student. Explain why you have chosen to work in this subject area and what it is about your proposed research that you find particularly exciting, interesting or important. Also explain the potential impact or wider benefits to society of your research. 

EDIT AFTER FEEDBACK:
Imagine you are standing on the rocky shore of a lake with a friend, who challenges you to a competition: who will be first to hit that floating branch with a rock? You and your friend hunt for suitable rocks to throw, hefting them in your hand until you find some that feel just right. You then take turns throwing those rocks to try and hit the branch, getting closer on each throw until finally one of you scores a direct hit.

The simplicity of this commonplace game is deceptive. Throwing is actually so difficult that humans are the only species who have specialised in it. We have hands that can grasp objects; bodies that help make the precisely timed throwing action possible; and the ability to perceive which objects are most suitable for throwing. Evolutionary biologists believe that throwing provided our relatively small and weak ancestors with the ability to hunt large animals such as mammoths. But throwing still plays a large role in modern life, even if it is more likely to show up these days in the sports arena than on the plains of North America.

Thanks to sports science, we know a lot about what throwing looks like when it happens. What we don’t know is how a skilled thrower produces that throw in the first place. What makes an object ideal for throwing, and how do we perceive this when we heft that object? How do we perceive where our target is? And, most importantly, how do we use this information to select a throw that will move this object in just the right way so as to hit that target? This project is all about answering those questions. We will measure expert and novice throwers aiming to hit distant targets and track their movements with high speed cameras. We will then analyse this data using cutting edge techniques to identify how people produce the throws they do.

I find throwing fascinating because there are so many unanswered questions from many different disciplines. Psychologists want to know how we perceive the environment and produce the throw; neuroscientists want to know how our brain works to support this behaviour; evolutionary biologists want to know how all these skills came together in our species and how this might have supported the evolution of language. My colleagues and I have the tools we need to actually answer these questions, and we believe that the answers will be of interest to many people, including scientists but also coaches and athletes interested in improving their techniques and accelerating their learning.

2495 chars

ORIGINAL
Imagine you are standing on the rocky shore of a lake with a friend, who challenges you to a competition: who will be first to hit that floating branch with a rock? You and your friend hunt for suitable rocks to throw, hefting them in your hand until you find some that feel just right. Then you take turns throwing those rocks to try and hit the branch, getting closer on each throw until finally one of you scores a direct hit.

This game might seem an odd thing to want to study scientifically – it all seems so straight forward. But this simplicity is deceptive. Throwing to hit a target like this is actually a surprisingly difficult action that requires you to produce a precisely timed movement that suits the task at hand (how far is the target? How big?). Throwing is in fact so difficult that humans are the only species who have specialised in it. We have hands that can grasp objects, bodies that help make the precisely timed throw possible, and the ability to perceive which objects are most suitable for throwing. Evolutionary biologists believe that throwing provided our relatively small and weak ancestors with the ability to hunt large animals such as mammoths. But even today, throwing still plays a large role in modern life, even if it’s more likely to show up in the sports arena than on the plains of North America.

Thanks to research in sports, we know a lot about what throwing looks like when it happens. What we don’t know is how a skilled thrower produces that throw in the first place. What makes an object ideal for throwing, and how do we perceive this when we heft that object? How do we perceive where our target is? And, most importantly, how do we use this information to select a throw that will move this object in just the right way so as to hit that target? This project is all about answering those questions, using modern motion-capture and data analysis techniques.

I find throwing fascinating because there are so many unanswered questions from so many different disciplines (e.g. psychology, neuroscience, evolutionary biology). But even better, my colleagues and I have the tools we need to actually answer these questions! We believe that the answers will be of interest to many people, including scientists but also coaches and athletes interested in improving their techniques and accelerating their learning. It’s rare to find a research question with such wide appeal, and this makes it a very exciting topic for me to work on as a scientist.
2489 chars

What are the units that perception measures the world in? Firestone vs Proffitt

Perception is an act of measurement, and, like all acts of measurement, it needs a scale in order to be useful. Think about placing something on your kitchen scales; all that actually happens is that the object presses on the scale and the scale registers that something has changed by some amount in response (the location of a tray, for example). In order to know what that change means, the change is presented to us on a calibrated scale (by moving a needle around to point at some number, for example). The needle always moves the same amount for a given weight but the resulting number can vary (you might have an imperial rather than metric kitchen scale, for example). Without the scale, you can say that one thing is heavier than another by noting that it moves the scale more (this is an ordinal evaluation) but you need the scale in order to say what the weight difference is (the metric evaluation).

Visual perception measures the world in terms of angles; objects subtend a certain number of visual angles that depends on their size, distance, etc. Your thumbnail held at arm's length is about 1° of visual angle. You can get ordinal information directly from angles (the fact that one thing is closer/bigger/etc) but you need a scale to get the metric information required to use vision to control action. For example, you need to perceive how big something actually is in useful units in order to scale your hand size appropriately when grasping it; relative size doesn't help. One of the fundamental questions in (visual) perception research is, therefore, what are the metric units that the perceptual systems use to scale their measurements?

Dennis Proffitt has been studying this question for a long time and is in favour of task-specific, body-scaled units. His evidence comes from studies in which people perceive their environments differently as a function of their ability to act on that environment. Probably the most well-known example is the study that showed people judge hills to be steeper when they are wearing a heavy backpack (Bhalla & Proffitt, 1999). The idea is that the backpack will make traversing that hill more difficult, and when the visual system measures the slope, it scales its measurement in line with this perceived effort. The hypothesis is that this is functional; it's a feature of the visual system that helps us plan appropriate actions. 

Perspectives on Psychological Science recently hosted a point-counterpoint debate on this topic. Firestone (2013) reviewed the literature on this type of action-scaling in perception and concluded that not only do the data not really support Proffitt's account, but that this account couldn't work even in principle. Proffitt (2013) rebutted Firestone's arguments and defended his view. I'm interested in this because Proffitt is at least a little ecological, and the basic idea he defends is one I would defend as well (although not in the form that he proposes). So who won?

Replication will not save psychology

"Replication is our only hope." "No. There is another"

Psychology is big into replication these days. A lot of people think that a major problem with the field is that many important results have not been replicated, and that this is in part because journals don't like to publish replications (not original or sexy enough). 

I'm all for replication; it's part of good science. But I've never been that into the whole 'replication movement' that's kicking around, and the reason crystallised for me during a 4am baby feed: 

Being able to replicate a study is an effect, not a cause of good scientific practice. So the emphasis on replication as a goal has the whole thing backwards. We should actually be focusing on improving the experiments we run in the first place. If we run better experiments, the replicability will take care of itself.