Tuesday 2 August 2011

What Does The Brain Do, Pt 2: The Fast Response System

I want to continue thinking about the implications of the claim that the brain does not trade in representations. I'm not looking to defend this view here; we got into it a bit last time, I've talked about it here, and if you can't imagine what cognition without representation might look like, then you should read this post by Sabrina and then Radical Embodied Cognitive Science before worrying me with your lack of imagination. If you then feel like getting into it, Sabrina is tackling this topic in detail, beginning here.

If the brain isn't mentally representing, what is it doing? Last time, I got into the idea that the brain is part of a embodied cognitive system. It's in the middle of a rich information flow, with access to perceptual information about the world and ourselves, and it's a critical part of the action system, involved in our responses to that information. I talked about it as the fast response system in the set of inherent dynamical resources available for us to use to form task specific devices; I want to expand on that a little.

Fast Response System
The task specific device approach is primarily a methodology; instead of trying to study the perception-action system in general, you acknowledge that this system is too high dimensional, flexible and non-linear and instead focus on studying performance in well defined task spaces to identify how the system solves specific problems. It is also, however, how I think the system itself solves the problem. Right now, as I sit typing this, my perception-action systems are not trying to effect any of the other things I can, in principle, do. I am not, at this moment, actually capable of running. I have the ability to form a running device; but that, I think, is not the same thing. So the perception-action system resolves the 'degrees of freedom' problem by only trying to solve the problems at hand, using the resources available.

Such a system requires a) rich perceptual contact with the environment, and b) the ability to transition from one device to another, in response to changes in the environment. (This 'environment' can include internal states, such as goals and intentions - but I don't really know how to talk about these rigorously yet so I want to leave that for now.) The ability to switch the entire device between states takes time (inherent dynamical resources such as muscles and cardiovascular system all have inertia and particular response characteristics; change takes time, in other words). The nervous system doesn't have much in the way of inertia to overcome; it can, roughly, turn on a dime, and respond on a time-scale that will match the changing flow of information. It can therefore lead the formation of the next device.

One of the critical features of task specific devices is that they are, to a certain extent, functionally encapsulated. What I mean by this is that devices literally use some resources and not others; those being used are temporarily 'walled off' from those not being used, so that changes to irrelevant resources don't unnecessarily affect behaviour. This fact underpins the perturbation experimental method (which I've described here for action components, and here for perceptual components).

Which parts are 'walled off' into coherent systems changes over time as you go from being one device to another. You therefore need to be able to temporarily couple these components into systems with specific composition and organisation, and then dis- and re-assemble them into the next system. As I discussed last time, most of our inherent dynamical resources are quite stable over time (e.g. the link-tendon system; muscle composition), and the coupling between these systems is often physical and not malleable. Our entire body is innervated by the nervous system, however, and it's speed and flexibility is the mechanism that allows for the temporary coupling of disparate resources into stable devices.

Which device gets made?
There is, of course, the question of which device to make. Traditional motor programming type theories (such as Schmidt's 'generalised motor programme' approach) hold that you call up a set of commands for producing a class of movements, and, in response to cues from the environment, you set the parameters of that schema (how fast, how far, etc) to produce a specific movement. In essence, you must retrieve from memory a schema that contains what you've learned about that kind of movement, and only then adjust it. Sabrina discussed this topic in great detail here - this structuralist approach assumes that we must have stored a discrete object of some kind, which we can then tailor to suit the local demands. This approach is fraught with problems: how many of these can we store? How long does it take to find and retrieve the right one? How do we pick the right one? What sort of commands are stored in the programme, given how context sensitive movement is (moving to the same location in space can be achieved by a large number of different movements - the mapping is therefore many-to-one)? Given the more functionalist approach I am advocating, this is clearly not what I think is going on. So what do I think is happening?

I always think about batting in softball as an example of this. When you're going out to bat, you pick up the bat, and stand 'on deck' waiting your turn. In between pitches, we all swung our bats, warming up. When it was my turn, I'd get my feet planted and stand by the plate; I'd touch the far corner of the plate with the bat to get a sense of where the strike zone was and swing the bat a few more times from that stance. I'd also often let the first pitch go by, just to watch it. All of this was on the advice of the better players in our team.

What I was doing was generating information that was guiding the formation of a softball-hitting device. Swinging the bat generates information about it's inertial characteristics, which is the basis of dynamic touch (discussed at the end of this post on Chemero's book). I now have information about how the bat is changing my upper limb functionality, as well as information about what's required to move the bat (how long it takes, etc). Repeating this at the plate continues this flow of information in a more specific setting, driving the continued formation of the right kind of device. Watching the first pitch go by was also a way of sampling some task relevant information without yet trying to interact with it. My claim is that this is all happening in the moment; until I pick up a bat, I am not currently capable of swinging a bat. The job of the nervous system is to enable the formation of that device once I begin the process, and the specific device I form is the result of the structure of the flow of information and the kind of nervous system and body it's flowing into.

The job of the nervous system is not to store discrete representations with motor commands that produce a given response. Our skilled actions are exquisitely responsive to environmental demands and it's not clear that tweaking a discrete set of categorical movement types could possibly account for this. Instead, the nervous system is reshaping itself in response to the changing flow of information, and that new shape couples inherent and incidental dynamical resources into the task specific device you are capable of making that is related to the task at hand. The current form of the nervous system, like the current form of all our inherent dynamical resources, reflects how it is currently being used, and while there are, of course, identifiable changes you can relate to that use, those are not representations, and the only reason they are described as such is because representations are assumed; this kind of imaging result is not independent evidence for representation.

How good your device is, of course, depends on practice and learning, and what I haven't yet touched on is the other putative role for the brain - memory. Why was it that when I picked up a bat I began forming that device? Because that's what I had practiced. What was the consequence of that practice that remained so that I could use it again later? If I was a structuralist I would say 'procedural memory'; a thing, an object, that contained the results of that practice in some format. As a functionalist, I want to say 'perceptual learning'; through use, I have come to learn what that information means, and what it means has to do with swinging a bat to hit a softball. This is clearly not good enough, but it's next on my list to tackle, so hang tight :)

A Final Note: Part of the problem with which I'm struggling is finding ways of describing what the brain is up to in my terminology, without sounding like I'm merely redescribing what we already know. I think the brain is up to radically different things that representationalists, but often I'm talking about the same overall behavioural effect (like the softball bat) with no clear evidence that my different mechanism is actually the right one. Modern neuroscience has 'explained' many if not all of the kinds of phenomena I need to explain too, so I will eventually need to be able to distinguish these accounts empirically. But a) I can't rely much on the existing literature because it's framed incorrectly for me to use, b) I'm not a neuroscientist and won't be running more useful studies any time soon, and c) right now my goal is to start from the ground and work my way back up, exploring the consequences of my differing starting point (inspired by what I like most about Gibson, 1979). I'm not even close to completing this; these posts are just my thinking out loud, and so yes, I know I have no slam dunk. What I do need, though, is continuing feedback about the logic of my approach, and awkward questions about things you see as weaknesses continue to be very welcome!


  1. I'm having trouble distinguishing what you mean by perceptual learning. Do you really feel that, in your softball example, you get better at perceiving the ball (your incoming information becomes better the closer to your standing 3-count you get)? Or that, in a Bayesian optimal integration way, you refine your contextual representation about the way the ball is moving today in relation to the way that the ball has moved in all other days (i.e., refined your prior), which is combined with the incoming perceptual information from the current pitch to give you the best chance of a clean hit? If you are pulling for something like the latter, I doubt you'd get much disagreement from what you consider the 'representation crowd' in the field of motor control...

  2. Obviously it's not the latter. As far as I'm concerned, Bayesian approaches to cognitive psychology are the same old schtick with slightly fancier maths.

    So right. In the softball example, what I'm suggesting is that which device is getting assembled emerges from the ongoing flow of information, a flow which is feeding into a system that has had experiences with that information flow and has developed responses to it. Cognitive people would say that what I learned was a representation with content I could retrieve and use later on. I want to say that what I learned is how to respond in the presence of that information.

    The difference is in the status of this knowledge when I'm not using it. I don't think it's meaningful to say I currently (sitting at my desk) have the ability to hit a softball; that ability isn't stored somewhere for me to access on demand. I do think it's closer to say I have the ability to become something that can hit a softball, and that this potential isn't stored anywhere; instead, it's how I respond in the presence of the relevant information, thanks to all that practice.

    This is definitely something I'm having trouble articulating though. But I think it's where you end up if you start over without representation.

  3. But what does the practice buy you? Better tuning of the relevant muscles and 'nicer' temporal coordination of joints at the low-levels by virtue of having developed the musculature more appropriately. Performing actions (or watching others perform them) makes you understand the action better because you get a more accurate understanding of what is likely to happen. But I feel we could argue this point forever - these outcomes seem difficult to disentangle...

    I was speaking to some of ecological object lifting types who seemed to have some pretty sensible views on representations - something along the lines of 'sure, representations and context and expectations and the such can influence action, but the real question is do they guide our actions all the time'.

    Now if this is at the hub of Eco Psychology, I buy it all the way - some 'solutions' work best in some situations. But I really cannot see how a hardline view which wholly eschews or endorses representational context is tenable...

  4. Well, as I say at the start, if you can't imagine cognition without representation, read Radical Embodied Cognitive Science. Agree with it or not, Tony achieves his goal or laying out what such a thing might look like and that it holds together as an idea.

    Ecological psychology is inherently anti-representational. Any eco-psych type who hasn't given them up yet hasn't read their Gibson well enough.

    As to what it buys me..this is the thing I'm still trying to articulate. I have to account for the fact that you improve with practice (ie practice changes you somehow in the direction of improved performance). The representational suggestion is this improvement is the development of an internal model, or improved prior, or what-have-you. I need a way of accounting for the persistence of the effects of practice without representation, and it's the thing people have the most trouble with (it's like thinking about things in their absence, if all you have is perception).

  5. How do you (or EE Gibson) deal with perceptual learning, without recourse to representations?
    surely if you are better able to discriminate between stimuli, that must rely on some representation of them? I know EE talked about 'differentiation' but there was no testable mechanism suggested? Saksida developed a computational model - but I presume you eschew such approaches? And Hall at York has been working on associative (and non-associative, habituation based) models.
    I will go and read the Radical Embodied thingy - but does it deal with issues such as familiarity, habituation and so forth?

  6. Andrew, I really like this phrasing!
    "I don't think it's meaningful to say I currently (sitting at my desk) have the ability to hit a softball; that ability isn't stored somewhere for me to access on demand. I do think it's closer to say I have the ability to become something that can hit a softball."

    Gavin and Pam,
    There are two cores of ecological psychology, a theory of perception and a theory of development. The two are intimately connected at the start, divorce into two lines when James takes the perception part and Eleanor takes the development part. Due to historic happenstance, the developmental side of the story has not been developed to nearly the level of sophistication as the perception part of the story, and the two have not been reintegrated together in a major way.

    To tie this in better with Andrew's post... If we wanted to phrase things more neutrally, we would say that there are several types of learning, one of them being Perceptual Learning. Perceptual Learning involves improvements in the coordination between the organisms perceptual system and their environment, specifically changes in how the organism is attuned to ambient energy (e.g. information in the optic array). That sounds a bit mystical, but it really just means that the organism becomes better able to discriminate nuances in the way the world is. The softball example provides an ideal context for such talk because surely the trajectory of the ball from the batter's glove specifies EXACTLY what the batter must do to hit the ball, so that a sensitive enough person would - if her system was attuned properly - never miss the ball. The batter does not need to represent anything, she just needs to be sensitive enough.

    Keeping it neutral, one might then ask how much of the development of organisms (particularly modern, social people) this particular type of learning could account for. It obviously cannot help much in non-perceptual tasks, but it might do a good chunk of the work in any perception-action situation.

    Now the non-neutral part.... Ecological psychologists assume that most, if not all, activities of people boil down to perception and action. Admittedly, for some this is an act of faith, but for most this belief is better understood as a guiding scientific principle. For Andrew, as for many others in the field (like me), the metaphysical issues is a driving factor, but readers of the blog can ignore that side of the issue if it is distracting them.

    If Andrew wanted to frame his current task more neutrally he would say something like: How do we describe the role of the brain in enabling people to do those things that don't require representations? We could then leave the question of what, if any, tasks require representation for another day (and even then send the inquirers to Tony).


    That went off track a bit. One last thought though: There are different time scales of development and Andrew is coming to see one of the problems that Animal Behaviorists have been struggling with a lot over the past 20 years. This post deals with the question of how I change from being the type of body that does something this minute to the type of body that does something different in 10 minutes. When sitting on the bench, I am not the type of organism that responds correctly to information about how to hit the pitched ball; as I approach the mound I need to become that type of organism quickly. How does this quick process of attunement relate to the long-term type of attunement that occurs as I learn to hit better?

  7. Thanks Eric, nice context as ever.

    Pam: EJ has two books on perceptual learning; the most recent is fairly up to date and readable.

    The differentiation account: let me copy and paste from one of my PhD qualifying exams (which I really should publish on the blog sometime):

    Beginning with the ecological assumptions about the nature of the information available to the organism (as defined and described earlier), EJ Gibson (1969) defines a differentiation theory as one in which perceptual learning is considered as “an increase in the ability of an organism to get information from its environment, as a result of practice with the array of stimulation provide by the environment.” (p. 77). By this account, organisms learn to distinguish a “variable of stimulation” from the “mass of impinging stimulation” (EJ Gibson, 1969, pg. 77) and begin responding to that variable that was not previously responded to. The rest of the process of learning (EJ Gibson (1969)’s criterion of learning) is one of increasing specificity – perception becomes more specific to and in greater correspondence with the variable of stimulation.

    A simple example may serve here. Tau is a “variable of stimulation” – it is the perceptual information presented to the organism. This variable exists within a flux of other optical variables and noise. By EJ Gibson’s account, the task of a perceptual learner is to first differentiate tau from these other optical patterns, and then begin to home in on, or attune to, tau, so that a) tau is being responded to when it was not before, and b) the resulting perception given the presence of tau corresponds more and more to time-to-contact (e.g. Yonas & Hartman, 1993)

    Affordances and the invariant information they produce are critical parts, because you can only learn to differentiate things that are sufficiently stable, relative to the surrounding (optical) flow.

  8. How does this quick process of attunement relate to the long-term type of attunement that occurs as I learn to hit better?
    This, in a nutshell, is the question I don't have an answer for. This post covers how we might think about the 'quick' time scale; I'm still worrying about the 'long-term' time scale question.

  9. I'm familiar with the general ideas Gibson gives about PL - but I find it descriptive - not explanatory. How is it that we become attuned to tau, over time?

    To me, the activity of neurons is a representation. And the activity of neurons to a stimulus which has undergone PL will change, compared to a novel stimulus, or a familiar stimulus that has not undergone PL (we tend to do expt looking at mixed vs blocked exposure). This means that the representation has changed. Or you could argue that the neural activity changing is a 'tuning up' in response to the environment. I think this is part of the 'long-term' scale answer. You can avoid recourse to the word 'representation' by talking about patterns of neural activity.

    I was at ICOM5 yesterday, and enjoyed the session on MTL, where Graham and Saksida were trying to convince memory people that in order to understand role of MTL in memory, you have to understand role of MTL in perception. Seemed relevant to your general goal of trying to figure out what the brain is doing.

  10. How is it that we become attuned to tau, over time?
    In terms of neural activity? This is, of course, the $64 question. The ecological approach has been focused on establishing mechanisms at the behavioural level, ie identifying that such tuning is happening. There isn't, to my knowledge, much neuroscience on what's happening to support this, but this is partly my point: neuroscientists aren't trying to answer this question because they don't know (yet) that this is the question to be answered! I can't use the literature to answer your sensible questions because the literature has the wrong job description for the nervous system.

    EJ does describe in detail a behavioural level process for progressively becoming attuned to information and what that information specifies. It's not quite a mechanism, as you point out - but it's the first critical step, to lay out what such a thing might look like. Claire Michels and David Jacobs have published a lot of work looking at the process of learning in terms of exploring an information space; it's been a while since I've read this stuff but it's definitely an attempt to turn EJ's process into something you can tackle empirically. I'll think about this some more, this could be a useful next post.

    You can call neural activity a representation, but what does it buy you? Nothing but the trouble Sabrina is spelling out. Another commenter a while back mentioned another way to talk about this kind of activity here; nothing is being represented, but activity is being coordinated and controlled. I admit it's still vague :(

    How were they tying perception to memory? What was the link?

  11. link was MTL - amnesics with MTL lesions have problems with object perception. relates to
    Bussey TJ, Saksida LM (2002), “The organization of visual object representations: a connectionist model of effects of lesions in perirhinal cortex.” Eur J Neurosci 15(2):355-64

    There is some neuronal recording work that looks at changing reponses with experience, in a way that could be described as tuning up.

    This is an old review of the field: not read it in a long time though!

    Edeline, J. M. (1999). Learning-induced physiological plasticity in the thalamo-cortical sensory systems: a critical evaluation of receptive field plasticity, map changes and their potential mechanisms. Progress in Neurobiology, 57(2), 165-224.

  12. Pam says,
    "I'm familiar with the general ideas Gibson gives about PL - but I find it descriptive - not explanatory."

    Depending on what you are interested in, you might be spot on with this observation. As a science advances its concepts become increasingly nested - what is explanatory for one purpose is descriptive for another. If the question is something like "How did he just do that?!?" Then the Gibsons's sciences of perception and development are explanatory. If the question is "How does the brain support such processes?" Then the Gibsons's sciences of perception and development are descriptive, i.e., the thing to be explained is the attunement of the organism. There is no satisfactory explanation to the latter question that ecological psychology can offer, though there are many (like Andrew) working on the problem.

    So, if it seems to you like Ecological Psychology stops right were you start to get interested, you are probably accurate. As Andrew said, right now in our field, that is one of the $64 questions. (My inflation is a funny thing, I would have said million dollar question!) The one obvious thing is that if you start with James and Eleanor's work as the DESCRIPTION, then the explanations offered by cognitive neuroscience are clearly unsatisfactory.

  13. The one obvious thing is that if you start with James and Eleanor's work as the DESCRIPTION, then the explanations offered by cognitive neuroscience are clearly unsatisfactory.
    Amen. This is precisely what I'm trying to get at, nicely put :)

    I also want to move on with positive suggestions, though, and for that I really need a friendly neuroscientist. Where can you go to publish/talk about challenges to neuroscientists? I have no neuro data but a clear behavioural research programme grounded in theory and lots of data which I want to present to neuroscience as an alternative grounding for doing business - where the hell do you send that paper?

  14. I'd try one of the Frontiers journals if I were you (Frontiers in Human Neuroscience if you really want to make sure you bump into the neuroscience crowd). The reviews, I gather, will also be published along with your manuscript so it might be a nice extension of the discussions on the blog.

  15. Not a bad idea. Now to find the money to pay for that open access goodness :)

  16. I am writing a chapter for a book on "Neuropragmatism", in which I will beseech for a better way to talk about what the brain does. I will give a summary of past work that seems in the right direction, and give some ideas of future directions. If you were interested in collaborating on that, it might be a nice venue in which to test run ideas as you continue to move forward -- i.e. it will not be an ideal place to publish definitive views, but quite a good place to put out preliminary ideas.

    Some authors seem enthused about opportunities such as this, others do not. While I personally admire those who work long hours without reward to produce definitive works, it seems prudent to me to take opportunities to publish bits and pieces on the way.

  17. P.S. Tony is also preparing a chapter.

  18. A book chapter could be very useful. Send me details of what you have in mind :)

  19. Just saw this. If you haven't read Cisek and Kalaska, do so! It is fantastic http://www.cisek.org/pavel/Pubs/CisekKalaska2010.pdf

  20. Interesting - I look forward to finding some time to read that properly!