On why fMRI is bullshit, even when you're doing it right

I'm just back from the 11th European Workshop on Ecological Psychology and I have a few posts about things that arose there. Ecological psychology is, reassuringly, in better empirical form than I had been thinking; there are still  people doing things badly, but plenty going after things as carefully as they should be, with due attention to issues of information. So that was good.

There was one talk by John Wann on the neural control of steering in driving tasks. I like John a lot, and respect his work - he's a very careful experimentalist and not at all susceptible to fads in psychology, and his work has always included detailed use of perceptual information in sensible ways. He has recently been involved in some fMRI versions of his steering studies with a post-doc trained in imaging, and he presented this data to the conference.

One of the things I like about John is the care with which he does things. I mentioned in my previous rant about fMRI that one of the big problems with these studies is that the researchers have no theory of information: they don't understand what they are pumping into the brain, and therefore can never actually interpret what happens in the brain as a result. John is the exact opposite of this; he uses optic flow displays rich in information but rigorously controllable, and as a result is able to run an fMRI study well.

The key to fMRI is comparison. You can't simply measure the brain activity at a given time, because even the simplest task makes most, if not all of the brain light up. What you can do is measure the activity under conditions X, then under conditions X+Y, then subtract one from the other to establish the neural correlates of Y. While this is essentially Donderian subtraction and subject to the same basic critique (that it assumes independent, non-overlapping processes and fails to the extent that this is violated) it is not necessarily a bad plan, and you can help refine your estimates by having other cases to compare, so as to triangulate the activity unique to Y.

If you don't have a theory of information, then you run the risk that what you pump in at X+Y is actually A+B; see my previous rant again for my tale of the biological motion perception study. In John's study, this is very much not the case. He is able to present observers with optic flow fields and add or subtract genuine sources of information, and this allows him to, at the very least, be running a valid study. The net result is some specific activation in an interesting part of the brain that can be tied to variation in behaviour during the task and related to the information content of the displays.

This was all fine, and I was very impressed: but I was left with one question (which, to be fair, is the one question John also tries to keep in mind too): so what? These displays comes from the detailed behavioural studies he and his colleagues have run over the years, and on the basis of those studies they have come up with a pretty solid model of what the relevant information is, how and way it's detected and how it's used to control steering (although it's not the only one; see Fajen & Warren, 2003). These differences in behaviour now have a neural correlate; but so what? Knowing where in the brain the relevant information was showing up doesn't seem to add anything at all to the account: they already knew what the information was, and that behaviour varied as a function of the accessibility of that information. The fact that a bit of the brain also behaved differently as a function of the accessibility of that information is, well, just not that surprising; after all, clearly the brain is involved in all this.

John, good scientist that he is, has asked himself this question and convinced himself that these data do add a piece to the overall puzzle, acting as supporting confirmatory evidence for the behavioural differences. He doesn't claim it's better evidence, or a more real effect, just that it's a part of the puzzle; and at one level I can't get too worried by that. And, to be fair, the data did potentially come in handy at a later talk looking about how patterns of brain activity in infants varied as they learned to understand what looming means (objects that are approaching you appear to 'loom', i.e. grow in apparent optical size) - one of the bits lighting up did seem related to the bit that John is now fairly sure is involved in the perception of precisely this kind of information in steering. If true, this uses data from a task where adults were able to respond in a meaningful way to interpret neural data from an infant who couldn't; it's inferential, but that can be ok. I wasn't that convinced, but the general idea is sound.

But all this has mostly just reaffirmed my view that fMRI just isn't worth the huge amounts of money it costs. Psychology is a field where adding expensive imaging in where it isn't needed is seen as a sensible way to help gain funding, and I'm more convinced than ever that it simply isn't helping. John's careful experimenting showed one thing: that he had already gained all the necessary data to produce a good model without the brain data, which, if it did anything, simply confirmed what he already had good reason to believe.

References

Fajen, B. R.,& Warren, W. H. (2003). Behavioral dynamics of steering, obstacle avoidance, and route selection. Journal of Experimental Psychology: Human Perception and Performance, 29, 343-362. DOI: 10.1037/0096-1523.29.2.343

Field, D.T., Wilkie, R.M., Wann, J.P. (2007).  Neural systems in the visual control of steering.  Journal of Neuroscience, 27,  8002– 8010. doi:10.1523/JNEUROSCI.2130-07.2007