Peer Review: Solemn Duty or Merely Extra Work?

The other day on Twitter I saw Rolf Zwaan tweeting about Collabra's policy of rewarding reviewers with credit that can be traded in for credit for organisational article processing fees (APC) budgets, into Collabra's fee waiver account, or for cash to the reviewer. This idea, of paying reviewers for reviewing, comes up now and again on Twitter and it always sounds wrong to me. My gut response is that we should not be paying individuals money to review articles.

So I did the normal thing and posted a Twitter poll, with the following results:

Thesis: getting paid for peer review is a bad idea because it's part of our collective duty to improve science

— Andrew & Sabrina (@PsychScientists) April 13, 2016

Turns out I am in the minority! The various discussion I've had around this tweet have been kind of interesting too. 

The Art of the New Collaboration

I love working with other people. It keeps me connected to a wide range of topics and drives me to push my work in ways I would never other think to do. I've had many wonderful conversations with people over the years in which we've thrown around ideas for collaborative projects. Often, however, these simply never get off the ground and I've struggled for years to figure out a way to turn good ideas into genuine collaborations. 

I have two new collaborations on the go right now and these are finally working, so I thought I'd write down what I think I finally did right that got these moving. I'd love to hear your thoughts and ideas about how to start new collaborations; I suspect there are many scientists out there, young and old, who would be very interested in what we think works.

My main thought is this. A new collaboration is, by definition, a new project that brings together the skills and resources of at least two groups. These two groups will know their stuff, but not necessarily the others, and that means there's work to be done bringing people up to speed. In addition, each group will have existing interests and projects, and that means finding a way to implement some new, probably unfunded work using what you have to hand without interfering with the existing work. 

My current solution that seems to be working: choose ONE simple, easy to implement project and work it through to completion. 

Affordance-based control (Fajen 2005, 2007)

The most commonly studied tasks in the ecological approach involve the perceptual control of actions such as interception and steering. These models all involve perceiving some variable and moving so as to null the discrepancy between a current value and an ideal value. However, none of these approaches involve the perception of affordances; specifically, none of them address how people work to keep the required corrections possible, given their action capabilities. Fajen (2005, 2007) proposes affordance-based control, an ecological research framework that brings these questions to the fore and leads to the discovery of new, affordance based control strategies that account well for the data and solve the problems of simple information-based control models. 

My current sense is that Fajen is absolutely correct in his assessment of the problems and has done sterling work developing an ecological solution. What follows is a brief description of the problems and his solutions; in the future I will blog some thoughts as I work to align my throwing affordance work with this framework.

Quantifying the Affordances for Throwing for Distance and Accuracy

I have a new paper in press at JEP:HPP (Wilson, Weightman, Bingham & Zhu, in press; supplemental material). It is the end result of five years work across two jobs, and it has involved kinematic data collection from expert throwers in Leeds and Wyoming, analysis of that data, then interpretation of that data in the context of detailed simulations we ran in order to identify the affordance property of the target structuring behaviour. This is my first paper on affordances, my first about my current favourite topic of throwing, and probably the heftiest empirical piece I have ever done, so getting it published in my journal of choice is pretty exciting!

I'm going to just lay out the basic framework of the paper here. I will leave the (very many) details to the paper. The paper consists of two experiments, a series of simulations, and a discussion of affordances as dispositional properties of tasks best described at the level of task dynamics. This last bit feeds into the argument in the (mostly philosophical) literature on the nature of affordances; bad news, people who think they are relations - they aren't, and I've got two experiments that back that up!

What Would It Take to Refute Radical Embodied Cognition?

People often send us papers and data via Twitter that they believe rule out a radical, non-representational theory of cognition. Because I have yet to agree about any of these studies, these people then often ask in exasperated tones 'well, what would you accept as evidence?'. 

My current best answer is "about 20 years of hard work". 

It's Time to Relabel the Brain

Another day, another study finds that 'visual' cortex is activated by something other than information from the eyes:

A research team from the Hebrew University of Jerusalem recently demonstrated that the same part of the visual cortex activated in sighted individuals when reading is also activated in blind patients who use sounds to “read”. The specific area of the brain in question is a patch of left ventral visual cortex located lateral to the mid-portion of the left fusiform gyrus, referred to as the “visual word form area” (VWFA). Significant prior research has shown the VWFA to be specialized for the visual representation of letters, in addition to demonstrating a selective preference for letters over other visual stimuli. The Israeli-based research team showed that eight subjects, blind from birth, specifically and selectively activated the VWFA during the processing of letter “soundscapes” using a visual-to-auditory sensory substitution device (SSD) (see www.seeingwithsound.com for description of device).

There's lots of research like this. People are excited by mirror neurons because they are cells in motor cortex that are activated by both motor activity and perception of that motor activity. It's incredible, people cry - cells in a part of the brain that we said 30 years ago does one thing seem to also do another thing. How could this be??

I would like to propose a simple hypothesis to explain these incredible results and that is that we have been labeling the brain incorrectly for a long time. The data telling us this has been around for a long time too and continues to roll in, but for some reason we still think the old labels are important enough to hold onto. It's time to let go.

I'm going to go out on a limb and say it's a bit more complicated than this

Psychology's real replication problem: our Methods sections

Replication has been a big topic in psychology recently as a) we've suddenly realised we need more of it and b) because there have recently been several high profile replication efforts published (e.g the Many Labs effort; see Ed Yong's summary). Last week Simone Schnall, one of the authors whose work failed to replicate in that project, wrote a blog about her experiences getting replicated. She has issues with the replication (specifically that their data had a ceiling effect on the morality measures which would obscure any differences) but one particular comment caught my eye:

Of course replications are much needed and as a field we need to make sure that our findings are reliable. But we need to keep in mind that there are human beings involved, which is what Danny Kahneman’s commentary emphasizes. Authors of the original work should be allowed to participate in the process of having their work replicated. (emphasis mine)

This idea that there is somehow a requirement to involve authors in efforts to replicate their work. This is nonsense; once you have published some work then it is fair game for replication, failure to replicate, criticism, critique and discussion. In other words, we're all allowed to science the hell out of your work any time we like. We don't need either your permission or your involvement: the only thing we (should) need is your Methods section and if you don't like this, then stop publishing your results where we can find them.

Of course, getting the original authors involved can be very productive; you can chat experimental details, make sure you have covered everything, and generally be collegial about the whole thing instead of adversarial. But there is no obligation to do this, and I'm surprised that people think there is one.

Replication will not save psychology

"Replication is our only hope." "No. There is another"

Psychology is big into replication these days. A lot of people think that a major problem with the field is that many important results have not been replicated, and that this is in part because journals don't like to publish replications (not original or sexy enough). 

I'm all for replication; it's part of good science. But I've never been that into the whole 'replication movement' that's kicking around, and the reason crystallised for me during a 4am baby feed: 

Being able to replicate a study is an effect, not a cause of good scientific practice. So the emphasis on replication as a goal has the whole thing backwards. We should actually be focusing on improving the experiments we run in the first place. If we run better experiments, the replicability will take care of itself. 

Embodied cognition in practice - some thoughts and an open invitation

Since our Frontiers paper on embodied cognition came out, Sabrina and I have been giving talks and fielding questions in emails about what we're trying to achieve. People first ask us 'why should we do this?' and the answer, from the paper, is because it works really really well. Research in this field has produced extremely powerful explanations of behaviours with extraordinary predictive power; no more small effect sizes plus a successful research programme for as long as you want to run it! What more reason could you need?

The question we then get from people who have become interested in using what we've done is 'How can I apply this approach to my research?'. The paper itself describes four key questions that we think are critical for guiding good experimental practice in psychology and cognitive science:

1. What is the task to be solved?
2. What are the resources available to solve the task?
3. How might these resources be assembled so as to actually solve the task?
4. Does the organism actually do what your described in 3?

We think this is really just good practice. The basic question in psychology is 'why did a person behave like this rather than like that?' and answering this means figuring out what the task is, from the point of view of the person doing the behaviour. We think these questions will help keep psychologists on the right track, and not get distracted by hypothetical mental entities like theory of mind and object concepts until the evidence demands them. 

We give two examples of this method in practice (the outfielder problem and the dynamical systems model of the A-not-B error), but people are still left wondering how to apply this to their niche. We want to help here; obviously we want people to start working this way and also we want people across a variety of topics to start working this way, so we can finally start accumulating some results outside of perception and action type tasks. 

I want to do two things with this post

1. I want to describe the mindset you'll have to get into to start doing embodied cognition in your field, which I hope might make our steps and our examples make a little more sense.
2. I want to open an invitation to people to post questions in the comments. We're busy and so I can't promise an immediate response, but if you have a task and you want to start thinking about it from our embodied perspective, post a comment with some details and we'll see what we can do to help.