Tuesday 30 March 2010

On why fMRI is bullshit

If you want to really confuse a psychologist, tell them you don't think there are mental representations mediating behaviour. Try it - they will simply assume you must be joking, because it has never occurred to them that it might be true. This, unfortunately, is the single biggest stumbling block in talking about Gibson to cognitive psychologists, because one of the radical ideas in ecological psychology is that there isn't any need to invoke representations.

The most common counter argument comes from modern neuroscience (specifically, neuro-imaging, and in particular functional magnetic resonance imaging (fMRI)). There must be representations, your cognitive friend will cry - we've seen them via fMRI! If I present a stimulus to a person in a magnet, you can literally see the brain light up. Clearly, the triumphant cognitive type will claim, Gibson is wrong when he says there are no representations because the brain is obviously up to something.

Well, yes - of course it is. The brain is a complex and expensive piece of biological equipment, so I certainly hope it's up to something. But this argument (and yes, I really have had this argument) simply rests, like all cognitive psychology, on the assumption of mental representation. The brain is up to something; mental representations are required; therefore the brain is representing.  Ecological psychology is therefore often accused of claiming that the brain isn't interesting, but this error comes from conflating 'brain activity' with 'mental representation'. Ecological psychology actually has yet to say anything much about the brain, but this is mostly because we are still figuring out what it might possibly need to be doing.

The fascination with the brain is almost entirely down to the development of fMRI techniques. Brain imaging with PET and EEG lacked the kind of spatial resolution required to start talking about which bit of the brain might be responsible for doing what kind of processing, but as soon as fMRI hit the scene cognitive neuroscience became the place to be. fMRI is convincing to people: look, they say, this bit of the brain really does become active when I present my stimuli, so it must be doing the things I think are required to solve my task. Interestingly, however, people will say this without ever actually thinking about what it is they just said and they will forget to question whether they understand their task at all.

Ecological psychology, when done carefully, is about solving this problem. Our research programme is to figure out the actual job description for the brain - remember, the assumption of mental representation is only justified under an assumption of poverty of stimulus. If there is no such poverty, then the job the brain needs to do becomes radically different (e.g. it no longer needs to enrich perception with prior knowledge). This is the reason why modern neuroscience isn't a slam dunk refutation of Gibson: none of their research correctly characterises the information in a task and so therefore they cannot possibly correctly interpret the brain's response.

This is a tremendous problem for neuroscience. They have managed to successfully present some kind of information in a format the nervous system is capable of responding to (light via the eyes, sound via the ears, etc). Given that the nervous system can detect it, it does respond; but if you haven't correctly described what went in, you cannot possibly interpret what came out.

Am I being picky? Is this just a problem for bad fMRI research, and people who really know what they're doing really do control their stimuli? I wish that were true, but it isn't. Neuroscience is currently premised on the idea that all the interesting work of our mental life is done inside the head; remember, visual perception begins at the eye for these researchers, and not out in the world in the interactions of light with surfaces as Gibson realised. Neuroscience is trying to explain everything about perception (it's contents, it's relation to things in the world) in terms of brain activity, and have simply failed to notice that a lot of the required work is actually not done anywhere near the brain at all. The optic array's structure is revealed by movement, for example; by eliminating movement (necessary for fMRI imaging) you've simply removed a large part of the information generally present when solving the task in question. The consequence is not simply 'controlling for movement', the consequence is 'fundamentally altering the task space'. This raises profound questions about the validity of fMRI research, i.e. it is not at all clear that researchers are even asking the question thinking they're asking.

This fundamental flaw in neuroscience is nicely illustrated by a story I tell a lot. When I was nearly finished at IU, the department was about the purchase an fMRI magnet and hire an imaging specialist. One of the job applicants was a high profile, in-demand researcher just completing a prestigious post-doc at NIH. This person was being flown all over the country by departments desperate to hire him because of how awesome everyone thought he was.

I went to his job talk, which was on some work he had been doing on the neural systems underpinning the perception of biological vs. non-biological motion. This is an interesting question: biological motion perception is a fascinating research topic that investigates the kinds of information we can gain simply from the way biological agents move (and it's a lot: age, gender, emotional state, etc etc; when done properly it is brilliant stuff). The important word in the phrase 'biological motion' is 'motion'; all the information is in the way things move in relation to one another, hence a point-light display can provide all the information with no confounding information.

Anyway, this job talk got off to a rocky start when he introduced his stimuli. His biological motion displays consisted of videos of someone dancing. Biological motion? Sure, but unnecessarily under-controlled, especially given that point-light displays are so readily available and solve all the key problems.

Then it got worse. His non-biological motion displays were videos of someone's hand holding a hammer and hitting a nail, with the hand digitally edited out. No hand = no agent = non-biological, for this person; this is, of course, entirely incorrect, because the video preserved the motion information. My PhD advisor and I looked at each other as if to say 'Do you want to tell him?', then he pointed out to the speaker that both of those displays were, in fact, poorly controlled biological motion displays; had he not thought to use point-light displays? The fMRI wunderkind had not, because he didn't know what those were.]

Let me restate this: a researcher studying biological motion perception had never heard of the single most important methodological tool for studying biological motion, and yet everyone thought he was fabulous because he had access to a scanner. [Edit: I misremembered this bit and it's not true; see the comment here.]

The worst part of this talk? He got differences in brain activity for the two stimuli type and proceeded to claim that the difference reflected how the brain processed biological and non-biological motion in different parts of the brain. And no one in the room besides my advisor and I thought this was wrong, even though the key flaw in his study had already been pointed out; he got differences, so the stimuli were clearly different, and the difference had been called biological vs. non-biological, so that is what the difference must be. The brain had spoken!

I tell this story to illustrate a few key points. First, fMRI research in general has no theory of information and therefore cannot be trusted to correctly interpret their results. Second, a room full of smart people believed the imaging data even when a fatal flaw in the method had been identified. The objection was ignored mostly because it came from the Gibsonian guy, and we all know those guys are crazy people who don't think the brain does anything! Third, this last fact tends to lead to fMRI research being sloppy with its stimuli in a way visual psychophysics research pre-imaging would never have found acceptable; pretty pictures have blinded psychologists to important questions of methodology and validity in a way that is simply shocking.

fMRI came along at precisely the wrong time for psychology. The dominant cognitive approach assumes that all the work of mental life is done inside the head by representations; imaging reveals that the brain is, indeed, up to something; and in a depressing example of the confirmation bias cognitive psychology triumphantly announces that fMRI has shown us the truth of their claims. Ecological psychology, dynamical systems and the various flavours of embodied cognition were only just starting to come together and demonstrate empirical successes, and we now have an uphill battle ahead of us.

32 comments:

  1. I want to start off by saying how much I enjoy these posts - understanding ecological viewpoints has been a goal of mine for some time now, and I'm enjoying rather easier to comprehend explanations and especially the context you are providing. That being said, I feel that the best way to understand something is to get to grips with its critiques. So I'm going to play some devil's advocate here.

    1. I feel you have set fMRI up as a bit of a straw man here - sure, if what goes is is not properly controlled, what comes out will be equally variable. But such is the case with the vast majority of experimental psychology. I make a dot appear on the right hemifield, and I have manipulated attention. Make someone coordinate their limbs with funny timings and I have manipulated difficulty. But this is probably as good as we are going to get, as long as we feel it appropriate to attach semantic labels to our manipulation, and attempt to describe our findings in pseudo-metaphor to make them comprehensible (x will alter the direction of attention, people will learn how to do y with practice). Describing behavior is a less thorny issue, since you can get around that (and ecological psychology gets around this by only describing behavior). But, as the all-mighty brain does indeed appear to control behavior, striving to pick apart the black box does not seem like a wholly ignoble task (once again, only as accurately as our labeling of the inputs and the outputs). And I'm not even sure it is totally unhelpful if you get the labels wrong. Differences in brain activation still, to my mind, point to differences in task - differences that are non-trivial enough to require measurable different activation with a pretty clumsy and underpowered measurement device (fMRI). The labels you ascribe to the differences are, for (depressingly) many, a function of what is sexy enough to get that nature paper. Does the manipulation have enough face validity to get that Nature paper?

    to be continued...

    ReplyDelete
  2. Of course validity is an issue for all science, and it's hard in psychology. I'm not going to deny that.

    But I have come to feel that fMRI has a fundamental flaw built into it; that flaw is the assumption that the brain is where all the interesting action is. The anatomy of the hand implements a lot of the 'required' computations to solve prehension; many of the problems facing the visual system are simply solved by movements; and so on.

    Coordinated rhythmic movement is a good example. It's quite a simple, stripped down task dynamic, but it's a complete task dynamic (i.e. it contains all the key elements). You get experimental control without removing key aspects of the task at hand. fMRI, on the other hand, is always an incomplete task dynamic, because it requires no movements. It will therefore always be asking the wrong question.

    Now, this isn't necessarily a disaster. I think it is probably possible to do good fMRI work if you acknowledge these flaws explicitly in your design and have a theory of information to work from. But no one does this to my knowledge (well, I do vaguely recall one guy at Brown who my advisor rates highly because he does understand information).

    fMRI is a tool, and it is regularly used beyond the range over which it applies. And holy shit am I sick of people thinking fMRI work automatically trumps behavioural work.

    ReplyDelete
  3. Incomplete task dynamic or no, that fact that people are able to actually accomplish the task behaviorally within the scanner works for me. Sure - I can derive the distance something is from me by how its pattern changes over time on the retinal array, but I can also do it when not moving. Of course, means I am doing the task with incomplete information, but isn't this just the same 'incompleteness' that people will encounter moving in a dimly lit room? In other words, aren't the vast majority of things the brain has to do 'incomplete tasks' in some way or the other? And while the lack of movement in the scanner is a pretty specific incompleteness, the eyes still move (real task saccades, or just microsaccasdes while fixating), and, as far as the observer is concerned for perceptual tasks, relative movements is all that matters.

    I too am sick of the assumption that fMRI trumps behavioral work. High profile publications are mostly payed for with the assumption of financial and time costs, rather than advancement of science. "Wow, what a lot of effort that publication must have taken!"

    ReplyDelete
  4. Sure - I can derive the distance something is from me by how its pattern changes over time on the retinal array, but I can also do it when not moving
    Not really, and why would you want to know what the brain is up to under such deprived circumstances anyway? It tells you nothing about normal function (I think) because all you've done is put people in a situation where none of the usual information is available and measured what happened in the brain. You've a) screwed up the task dynamic and thus aren't measuring the behaviour you thought you were studying, and b) still only measured what happened in the brain in response to impoverished stimuli.

    And actually, now I think about it, the flip side of all this is the response measures available. Sure, you can judge distance or size or whatever; but judging distance and actually effecting a reach-to-grasp movement are two entirely different things and the two measures are only slightly related. Given that we perceive things like distance in order to control action, measuring the brain activity while people view bullshit stimuli and judge distance while controlling nothing tells me nothing of any interest at all, about anything, especially perception-action.

    This is a key part of it that I didn't go into but should have, I think. There is no perception, and action; there is only ever perception/action. Talk about problems coming from labels: people see two systems when there is only one because there are two words. Perceiving-and-judging is as different from perceiving-and-reaching as two things that are entirely different from each other.

    ReplyDelete
  5. I guess the point I am trying to get at is that all 'normal' situations are likely to deprived in some way or another. Differences in lighting, occlusion, binocularity, coloration - the job still gets done. You recognise the stimuli placed in front of you. You know how far away it is. Perhaps you can't interact with it as well as you may in a more dynamic circumstance, but you have no real awareness of your potential difficulties in this sort of perception. Close one eye, cover my nose up, stand still, and you will still be able to recognise a picture of my face.

    I guess it was assumed for years (intuitively) that this perception subserves action, in a linear way. We perceive it, have awareness of it, think about it, act upon it. But, while this assumption is now questionable at best, the fact that we can passively perceive things without an intention to act upon them is something that I find amazing(ly wasteful?). And it is the apparent wastefullness of having 2 distinct (redundant?) perceptual systems that I think holds back a lot of the ecological acceptance (if I can do a task in conditions where ecological rules do not apply (?!?), why do I not use this strategy all the time? Or, why does the world not fade away when I am not moving/interacting?)

    ReplyDelete
  6. 1. Why not use it all the time? Because it is an attempt to solve the problem in a way that introduces insurmountable problems. Evolution certainly doesn't allow that sort of system to ever get a toe hold.

    2. Why does the world not fade away? Because you are never, ever still. Hold your body still and your eyes still bounce around. Fix your gaze and your eyes still flicker at 3Hz, not to mention the recent discovery of the endless and constant micro-saccades going on. And if you do remove motion (by placing an dot on a contact lens so that the dot doesn't move relative to the eye) then the dot will, indeed simply fade away.

    3. Various degraded situations: yes, these situations do occur. But they have consequences. Have people behave under monocular conditions and their behaviour will be pretty good but not as stable as with binocular vision because information detectable with one point of observation exists but is less informative than that available with two. Dim the lights, same thing: information becomes harder to detect. Behaviour follows the availability of information, not awareness (which plays no necessary role in causing anything).

    But clarify for me what you mean by 'perceive without an intention act on that perception'. I think I know what you mean, but I don't want to only address what I think you mean :)

    ReplyDelete
  7. I have been busy with the moving, but I'll try to reply to these posts shortly, to keep up the debate.

    ReplyDelete
  8. Very interesting points here.

    I'm still trying to understand your view on the subject.

    As a Medical Doctor I read in school about, for example, the motor cortex and it's spatial distribution, then when I encounter that hand movement can be "visualized" in the fMRI, the conclusion for me is that fMRI is a tool that can show how the motor cortex doing something.

    However, saying that the change in blood flow in certain areas of the brain during a task is equal to the "mental representation", seems far out to me and even dangerous (like phrenology).

    What do you mean with "fMRI lacks theory of information"?

    ReplyDelete
  9. fMRI researchers don't have a theory of what they are pumping into the system (actually psychology doesn't, because they haven't embraced Gibson yet). If you don't have a theory to describe accurately what you are pumping in (in terms of the stimuli) you cannot interpret what happens next.

    ReplyDelete
  10. I think I see what you mean, where you are saying that they (fMRI researchers) don't understand the stimuli or how it's perceived.

    From my point of view, if I tell a person to push a button inside the MR scanner every 2 seconds for 2 minutes, my theory is that I will see a significant BOLD signal in the Motor cortex (premotor probably as well).
    If I flash a light from time to time, I expect to see a significant BOLD signal in visual cortex and so on.

    So when you say fMRI is bullshit, do you mean that my assumptions in these cases are wrong?

    ReplyDelete
  11. I'm saying, what is it about the light you flashed that made the BOLD signal do one thing rather than another? If you can't describe ahead of time what is different about the light, you can't interpret any difference.

    On a related note:
    http://blogs.discovermagazine.com/notrocketscience/2010/04/01/scientists-discover-gene-and-part-of-brain-that-make-people-gullible/

    ReplyDelete
  12. I see, I see. Interesting.
    Well, as I said, I thought psychology had that figured out already :P

    I'll try to get that book you recommended, I hope it's for the psychology handicapped.

    ReplyDelete
  13. 1. It seems to me evolution has already allowed such a system to get a toe hold. The question is whether we use that system all the time, non?

    2. Yes, but this is not the movement Gibson talks about as being so vital for visual perception. And, given that you do make these eye movements in the scanner... I guess the point i was trying to make here is that you can complete pretty much any perceptual task within the scanner, lying down and static. I would acknowledge that your brain's solution to the particular problem may not generalize to other situations where the incoming information differs (with movements etc), but enough researchers use perceptual tasks that simulate movement on the retina with the correct geometries to simulate the visual gain of movement.

    3. How many situations is the system operating under perfect circumstances? Very few times, I would suggest. Of course, perfect is relative to what evolution has defined it to be for us, but there seems to be enough variability in our visual environments to make me think that, broadly speaking, what works in one situation will probably be what is working in another situation - a solution broad enough to cope with all contingencies (representations enhancing perception).

    and, i suppose if i did want to pitch this in a 2 visual streams kinda way, i'd say that the dorsal stream, immediate action sorta tasks are processed in an ecological sorta way, while ventral stream style tasks are optionally not processed in that sorta way. But surely someone must have thrown that dichotomy up somewhere along the way...

    ReplyDelete
  14. Actually I don't think that evolution has allowed such a system a toehold. I quite firmly reject the idea that just because you can get a response from the system implies that the system is built to respond to that stimulus. I'll think about this some more and flesh it out, but the assumption that 'the system provided a response under these impoverished conditions' means 'the system uses this variable we isolated in normal activity' is quite flawed, I think, and comes from a rather old school, linear way of thinking about this. That's not a brilliant answer, though, so I'll think some more.

    Gibson very much talks about how motion is not simply something that happens but is fundamental to vision. He knew very well that the eyes are in constant motion (and that they are in a head which is also always moving, etc) and recognised that this was only a problem if you assumed the movement wasn't part of the system's normal operation.

    Representations enhancing perception: my point is always to try and show that this is never necessary. If viewing conditions aren't perfect then you actually suffer the consequences, or move so as to improve the conditions, etc. But if you want to tell a story about representations enhancing perception, you need to be able to tell a story about when and how the representation is applied. What about viewing something moving behind a picket fence: it moves in and out of view but you can track it - are you alternating perception-representation-perception? (No, you are using perception, because perception does not merely happen in the instant, but that's a whole other thing).

    The 2 visual systems stuff would be more interesting if a) people could decide, for principled reasons, on what the two systems handle, and b) why you would ever want to separate function anyway? Every single time I hear the 2 visual system story, the two streams are supposedly responsible for different pairs of tasks; it's rubbish. It's information, not anatomy, that matters (as Geoff and colleagues have recently shown:

    Bingham, G.P. & Lind, M. (2008). Large continuous perspective transformations are necessary and sufficient for perception of metric shape. Perception & Psychophysics, 70(3), 524-540.

    plus Young's dissertation, unpublished so far I think but she only defended a few months ago). I'll blog about this later, Geoff is visiting and talking about this data).

    Lots more topics to talk about, clearly :)

    ReplyDelete
  15. Indeedy! I'll stop filling up this thread, and let you get on with the next topic though :)

    ReplyDelete
  16. Ha!

    Feel free to keep going, these are all thing I need to think about.

    ReplyDelete
  17. We will publish these discussions in a special issue of EBR or something. Old school style.

    ReplyDelete
  18. As the "wunderkind" mentioned in this post, I feel compelled to point out a factual inaccuracy. The post claims
    "My PhD advisor [asked] had he not thought to use point-light displays? The fMRI wunderkind had not, because he didn't know what those were."
    For evidence to the contrary, please see

    Beauchamp, M.S., Lee, K.E., Haxby, J.V., and Martin, A.: FMRI responses to video and point-light displays of moving humans and manipulable objects. J Cogn Neurosci 15: 991-1001, 2003

    ReplyDelete
  19. Hi Michael

    Thanks for stopping by, and thanks for the reference.

    I just flicked through the paper, and it's a lot tighter in it's design and interpretation than the data you presented in the talk. There's nothing in the paper about biological vs. non-biological motion, for example, the stimuli are talked about sensibly in terms of their motion properties.

    So that's good stuff, and I'm glad that I'm wrong about the point-light displays. In my defence, though, when Geoff asked you about point light displays you did say you didn't know about them; clearly just a misunderstanding but it was all I knew about you at the time. Plus your talk was about the neural correlates of biological vs. non-biological motion perception, and you described your tool stimuli as non-biological motion, which is incorrect, so I stand by that.

    A quick poke around, and it looks like the data are the result of Beauchamp et al (2002); is that right? If so it might be worth me reading that in detail, to see if I can still identify the problems I'm worried about.

    Beauchamp, M. S., Lee, K. E., Haxby, J. V., & Martin, A. (2002). Parallel visual motion processing streams for manipulable objects and human movements. Neuron, 34, 149–159. Download

    ReplyDelete
  20. The post claims that after a talk on Indiana University (on February 10th, 2004),
    "My PhD advisor [asked] had he not thought to use point-light displays? The fMRI wunderkind had not, because he didn't know what those were."
    I have posted the slides from this talk at
    http://openwetware.org/images/c/cd/IUTalk.pdf
    The second section of the talk (slides 48 - 57) discuss my point-light experiments in some detail, including crediting Johannson, the first to use point-light displays for psychological experiments. My slides demonstrate how we created our point-light displays using the Vicon motion capture system in the NIH Clinical Center and summarizing the key points of my Journal of Cognitive Neuroscience paper of the year before:
    Beauchamp, M.S., Lee, K.E., Haxby, J.V., and Martin, A.: FMRI responses to video and point-light displays of moving humans and manipulable objects. J Cogn Neurosci 15: 991-1001, 2003
    available at
    http://openwetware.org/wiki/Beauchamp:Publications
    and my Neuron papers, available at the same URL.

    ReplyDelete
  21. In that case I apologise; I clearly misremembered. I'll edit in a note above.

    ReplyDelete
  22. I would recommend reading advanced, or even introductory, sources in both cognitive neuroscience and neuropsychology before blogging about this topic. Such rudimentary research would have saved ADW and others this considerable humiliation.

    ReplyDelete
  23. BTW, we're aware of Mike's impressive work - he has the scientific kahunas to back him up. Let's see your body of work - a CV perhaps?

    ReplyDelete
  24. Hiding behind 'Anonymous' while demanding the credentials of the blogger seems pretty weak...

    ReplyDelete
  25. Not at all. It is necessary to identify the real "expert" here. Is it Michael Beauchamp (whose knowledge and life's work were both misrepresented in the blog) or Andrew Wilson (who wrote the blog trashing Mike's work and fMRI in general). The credentials (or identity) of anybody else is immaterial. Michael's credentials hold up.

    ReplyDelete
  26. One misremembered element doesn't make for considerable humiliation, although it was clearly an error and I did mean it when I apologised. It also doesn't really derail the point of this post. I don't argue from anecdote; this story was just an example I had of the point I was drawing. fMRI researchers, in general, have no clear theory of information and thus the results which emerge from their research are highly confused and prone to be interpreted in very odd ways. I stick by that.

    More relevantly, perhaps you might like to read this later post where I highlight that I am actually just deeply unimpressed by fMRI even when done by people I know and trust to do good science and who do, in fact, have a theory of information.

    Anonymous, if you look on the 'About Us' page you'll find links and information about me. I'm not a neuroscientist, but I am an expert in visual perception and it's role in the control of action, as well as being well trained in cognitive science from a good programme. I know enough to have an opinion about fMRI on a blog, I think.

    Feel free to point out where else I'm wrong, though, I'm happy to have the conversation if you have evidence you think I should see.

    ReplyDelete
  27. Psychology is trying to quantify qualitative data. I read this article, paraphrasing "Researchers lose credibility due to methodology". Do I have a degree in psychology? (No)

    ReplyDelete
  28. Thanks for your comments. I understand that 'Gibson' is a good source of reference on this matter? Can you please refer me to one of his books or articles, for further research into this subject? Thanks John Robinson!

    ReplyDelete
  29. I just stumbled across this post, and although it is 4.5 years later, I feel the need to comment.

    I am rather confused at your vehemence over fMRI. I was trained as a neuroscientist and use fMRI as a tool to study the brain. I often read papers and think that the authors are not clear in the theories behind why they are investigating something but that to me indicates that more rigorous science and methodology is at fault, not fMRI itself. I have also seen some terrible behavioral studies before, but that would not lead me to run screaming through the streets proclaiming that all behavioral researchers have no idea what they are doing.

    As someone who received a PhD in neuroscience, I learned about all aspects of the brain. Visual systems, neurons, that tricky, tricky biological motion. But fMRI research is performed by all types of scientists from all scientific backgrounds, and sure...not everyone who does this type of research comes from the same background and thus has the same skill set. That does not invalidate the research as a field.

    ReplyDelete
    Replies
    1. Hi Jillian, thanks for stopping by.

      I have calmed down a little since I wrote this :) That said:

      I still think fMRI is a deeply problematic technique that leads people to conclude lots of things that aren't true, and that interpreting the data depends strongly on your theory of cognition in general in ways people often don't admit or even know.

      Because of this theory dependence, I think fMRI will remain problematic until we get our theories organised (which we continue to work on here in the blog and in papers). I also think neuroscientists need to be more explicit about the theories that are informing their interpretations: the way in which cognitive theories and neural data can inform and constrain each other is actually very complicated.

      I also stand by the fact that fMRI studies are often (not always, but often) very average behavioural studies taking place in a magnet. Stimulus construction, the facts of being in a bore etc all have major consequences for what the brain will do in response and losing control of that is kind of unacceptable, especially given how expensive this research is.

      I am less down on neuroscience these days because our developing work is finally finding ways to connect our theory to the operation of the brain and I feel better now we're finally able to make some predictions that might actually be testable. It means that I now have some specific reasons to believe that the tools can get used in a way I think will produce good answers because the questions will be better (even if we end up being wrong, our questions are better :)

      Delete