Notes from Two Scientific Psychologists

Newton International Fellowships

2012-02-01T14:35:00.001Z

If you are looking for a post-doc opportunity in the UK, and are trained in perception, action or embodied cognition type research, then this is an excellent funding stream and we are both very interested in hearing from you to come and work in our labs. Please feel free to contact us if interested, and please spread the word to other interested parties!

*************************
A new round of Newton International Fellowships - an initiative to fund research collaborations and improve links between UK and overseas researchers - has now opened.

The Newton International Fellowships are funded by the British Academy and the Royal Society and aim to attract the most promising early-career post-doctoral researchers from overseas in the fields of the humanities, the natural, physical and social sciences. The Fellowships enable researchers to work for two years at a UK research institution with the aim of fostering long-term international collaborations.

Newton Fellows will receive an allowance of £24,000 to cover subsistence and up to £8,000 to cover research expenses in each year of the Fellowship. A one-off relocation allowance of up to £2,000 is also available.

In addition, Newton Fellows may be eligible for follow-up funding of up to £6,000 per annum for up to 10 years following completion of the Fellowship to support activities which will help build long term links with the UK.

The scheme is open to post-doctoral (and equivalent) early-career researchers working outside the UK who do not hold UK citizenship.

Applications are to be made via the Royal Society’s online application system which is available at https://e-gap.royalsociety.org/ The closing date for applications is Monday 16 April 2012.

Further details are available from the Newton International Fellowships website: www.newtonfellowships.org

Introducing Elliott Thomas

2011-12-25T14:21:00.002Z

Sabrina and I are proud to announce our most recent joint production: Elliott Thomas Wilson Golonka, born at 16:05 GMT on the 21st of December, 2011, at St James' Hospital in Leeds to the tune of 'Fairytale of New York' (seriously, you can't escape that song this time of year, even in operating theatres!). 8lb 13oz, all fingers and toes present and correct! Thanks to everyone at St James for their sterling work, and to the ongoing midwifery support courtesy of the UK's National Health Service. NHS, we love you lots!

Elliott will be be contributing his experiences as a developing perceiving acting embodied organism to the blog (and, most likely to another blog we'll set up to nerd out about our son). We look forward to his unique insights into the processes of perceptual learning and development, and to recreating some fun experiments in the comfort of our own home.

Merry Christmas and Happy Holidays to all our readers; 2011 has been a big year for us in lots of ways and while obviously the blog is going to slow down for the next while, we're looking forward to more fun in 2012.

Andrew, Sabrina & Elliott

Leaning to the left makes you believe odd things about embodied cognition

2011-12-14T11:33:00.000Z

Embodied cognition is not what you think it is. But I do understand why people think differently; it's because of the depressingly endless stream of papers published in Psychological Science that claim to have found that body posture somehow influences the contents of some cognition about the world. The latest "exciting" new finding claims that estimates of magnitude (size, amount, etc) are affected by your posture. The paper is well summarised at the Guardian for those without access to the paper (UPDATE: I am also talking to Rolf Zwaan, lead author of this paper, in the comments section there.)

It's a terrible paper, so it's apparently time for another in what might have to become a more frequent series, In which I am a bit rude about a rubbish paper and worry about how to kill papers like it. At the end, I've also talked a little about the role science journalism plays in maintaining the momentum for papers like this, via their own version of the file drawer problem. I'd be interested in people's thoughts.

The claim
Eerland, Guadalupe & Zwaan (2011) tested the hypothesis that we mentally represent magnitudes as a number line (Restle, 1970). This line runs from left to right; smaller numbers are to the left, larger numbers to the right. The suggestion was that leaning to the left should make smaller numbers more readily accessible, while leaning to the right should make larger numbers easier to call to mind; a priming effect. They tested this by manipulating posture while people made estimates of the magnitude of a bunch of things.

The questions were designed to be things people didn't know the exact answer to, but that they could generate a non-random estimate for. Example questions are "What is the height of the Eiffel tower in meters?", "How many people live in Antwerp?" and "What is the average life expectancy of a parrot in years?". While people were generating their estimates, they stood on a Wii Balance Board which tracked their centre-of-pressure (a measure of posture). They were instructed to stand 'upright', and to do this the data from the balance board was presented to them as a dot on a screen with a target - a computer warned them if their posture slipped away from the target. The mapping from posture to the screen was manipulated without participant's knowledge, so that in order to keep the dot on the target people were actually leaning left or right or were upright (varying by about 2%).

The question was whether this postural manipulation had any effect on estimates of magnitude. The questions all entailed different scales with different units (e.g. metres vs kilos), so to make the data comparable the authors z-transformed the results. This is a standard transformation; a z-score is equal to (data point - the mean)/the standard deviation, and the numbers that come out are unitless; they can be directly compared. Experiment 2 was the same design as Experiment 1 but all the questions had answers that ranged from 1-10 and participants knew their answer had to fit this range.

In both experiments, there was only a main effect of Posture, and comparisons revealed that leaning left made estimates smaller than standing upright or leaning right, but that leaning right was not different from standing upright. The authors conclude they have supported their hypothesis and that posture influences estimation of magnitude in accordance with the mental number line theory.

Why this is all entirely wrong
Where to begin? First, and I really can't seem to emphasise this enough, this is not embodied cognition. Second, the experimental question itself is ill-posed - at no point do the authors ask 'what is the function of magnitude estimation? What's it for?'. If we estimate magnitudes in order to actually do things in the real world, and these estimates are so fragile that they can be nudged by a 2% variation in posture, we would never be able to achieve our goals that depend on these magnitudes. So the paper itself is theoretically incoherent, yet another victim of psychology's lack of a central theory. Third, Restle's (1970) paper is not about magnitude estimation - it's about how people do fast addition of numbers. In addition, it claims that the mental number line is very fuzzy, with large regions being treated as the same; this is relevant to the discussion below about power and effect sizes.

But, just like the 'moving through time' paper (Miles, Nind & Macrae, 2010) the real disaster area is the experiment itself. If I had reviewed the paper, it would have gone something like this:

1. The hypothesis was clearly written once the data were in
The authors say 'We hypothesized that people would make smaller estimates when they leaned slightly to the left than they would when they leaned slightly to the right." (pg. 1511). They make no reference to the idea that leaning left and right relative to upright should produce different results, which is clearly implied by the mental number line theory.

2. The data don't actually support the mental number line theory
Leaning left produced smaller magnitude estimates on average across all questions and participants, but leaning right did not differ from upright. There's nothing in a mental number line that should make the left pull on behaviour harder than the right.

3. Handedness doesn't explain this
The authors do try to explain this asymmetry by saying that all their participants were right handed and there is evidence that "when people attempt to balance themselves, they show a subtle directional bias favoring whichever hip is on the same side as their dominant hand (Balasubramaniam & Turvey, 2000)" (pg. 1513). In essence, they claim that 'upright' for a right hander is actually slightly right of actual upright.

The reason this doesn't explain their results is that they utterly fail to test this. They recorded centre-of-pressure data, but do not analyse it at all and so they have no evidence of an actual shift. If they did have evidence of such a shift, they should have run a control condition where they identified each person's 'upright' position and calibrated 'left' and 'right' relative to that baseline (and a real journal would have demanded that they run this experiment; resolving a confound requires data, not a half-hearted citation). Actually, in the Methods, they mention calibrating the balance board for each participant, so either they calibrated it the way I just suggested (thus ruling out their handedness explanation) or they calibrated something else and it's not clear from the paper. Regardless, they don't get to call on handedness.

(A side note: if this handedness bias exists in their data, and if the lean influences magnitude estimates, then the bias is the only reason they found the effect to the left; it increased the size of the experimental manipulation to the point where it could be seen in the data. So even if the author's theoretical claims are true, this manipulation check suggests they were very lucky to see anything because they didn't manipulate posture enough).

4. There is no postural data in this paper
The authors report no data to support their manipulation of posture. Posture involves sway - we bipeds are dynamically unstable (like upright pendulums) and postural control is an active process of error correction. The authors report that the manipulation moved the centre-of-pressure by about 2%; but this will be on average, and given that the postural control system wants to keep you actually upright, there will have been plenty of sway back towards being in line with the pull of gravity. This time series of sway data should then have been indexed so that you knew which way people were swaying when you asked them the question.

As it stands, we have no way to evaluate how successful their postural manipulation was (which, given the manipulation failed in one direction, is critical information) nor what posture was actually up to when they answered the questions (which, given the next point, seems fairly crucial to me as well).

5. Many of the questions do not show the effect
The Supplementary data reports the median response for each question and sway condition. Only 25 of the 39 questions have larger numbers in the right sway condition compared to the left sway condition; only 9 show a pattern of increasing magnitude from left to right when you include the upright condition. The authors clearly got lucky in finding their result, suggesting the odds of replicating the effect are low.

6. The effect sizes are tiny
Backing up that last claim, I looked at the effect sizes. The authors report η² which is effectively 'proportion of variance explained by the effect'. The effect sizes ranged from 0.07 - 0.17; 7-17% of the variance was explained by the posture manipulation. These effect sizes are very small (in my experiments with 8-10 people I typically see effect sizes around .7 or .8). In other words, the statistical trend that enabled the main effect of posture to become statistically significant is very, very small, and only the large amount of data is allowing sufficient power to find it. This type of tiny effect is fairly common in psychology (hence the typical large n's in psychology research) but it makes many of us worry about statistical vs actual significance.

Summary
This paper is not an example of embodied cognition, and worse, there are methodological and analysis flaws throughout the paper that make the data fairly uninterpretable. But, just like Miles et al (2010) this effect is now in the literature and in the popular press and there's no obvious way to kill it. Replication is unlikely to work, and even if it did, snazzy journals like Psychological Science don't like publishing mere replications (or,worse, failures to replicate). These high-impact journals publish exciting results that never get looked at again and thus are never corrected in the literature; it's a serious problem with real consequences (e.g. Diederik Stapel's fraudulent work showed up in many of these high impact journals and was never subjected to the replication attempts which would have revealed the fraud).

A note to science journalists
The popular coverage of papers like this, and the general lack of interest in follow-up on the internet, helps make these ideas spread and never get corrected. The problems with this paper, however, are mostly the kind of thing you spot when you're a scientist with experience with the literature and the analyses. Science journalists typically do not have this expertise; their job tends to make them less specialised, so they tend to lack the specialist knowledge and broader context that makes these papers stick out like a sore thumb to a reader like me. I'm not passing judgement on this lack of specialisation - it makes perfect sense and it allows them to engage with a broader audience than I will ever attract. That's their job, and that's fine.

But what I would love would be if running their analysis past an interested scientist became more standard. We have our biases and foibles so you'd have to be able to take all the advice with a grain of salt and with one eye on the different goals of science and science journalism. In this age of 'research impact', where we are heavily evaluated on our ability to see our work have consequences beyond the 20 people who read our journal article, I'm willing to bet you could find plenty of people happy to give a paper and a write-up a once over and you could take our advice or not. Maybe this just runs into the various problems with peer review, and I'm depressingly aware that it would be fairly easy to get glowing reviews on this paper, for example (clearly they've already had a couple via Psych Science). But as we try to develop a sustainable and sustained research programme centred on our theoretical approach, these results create noise we have to fight through, and I am personally very keen to address these issues in front of a wider audience.

Another issue is science journalism's version of the file drawer problem. The big science blogs all write positive write-ups of the exciting news, because that's what they're trying to communicate. But not writing a more critical piece on something that's getting a lot of attention creates the same bias as not publishing failures to replicate. I wrote this post in part because no-one else was going to; science journalists, do you have room to ever do this kind of work? One good example may be the 'arsenic life' story, although that was still kicked off by a scientist who happened to blog; so maybe there is a use for us after all!

References
Balasubramaniam, R., & Turvey, M. T. (2000). The handedness of postural fluctuations. Human Movement Science, 19, 667–684.

Eerland, A., Guadalupe, T., & Zwaan, R. (2011). Leaning to the Left Makes the Eiffel Tower Seem Smaller: Posture-Modulated Estimation Psychological Science, 22 (12), 1511-1514 DOI: 10.1177/0956797611420731

Miles, L., Nind, L., Macrae, C. (2010). Moving Through Time Psychological Science, 21 (2), 222-223 DOI: 10.1177/0956797609359333

Restle, F. (1970). Speed of adding and comparing numbers. Journal of Experimental Psychology, 83 (2, Pt.1), 274-278 DOI: 10.1037/h0028573

Some Ground Rules for a Theory of Psychology

2011-12-09T11:23:00.000Z

Add psychology to the list

A fairly common response to our theory post was 'here's my theory, which is designed to replace and fix all the others'. However, it's more a symptom of the problem I was discussing than a solution for everyone to have their own entirely separate theory which doesn't talk to any other work in the field (see above). One of my personal goals in science is to not be that guy. I want to see cognitive science become more integrated, not more fragmented. We have also been asked, however, and quite sensibly, what we think the solution to our problem is. The question then is how to propose a theoretical approach for psychology and cognitive science where we don't just reinvent the wheel.

Sabrina and I have been working on this for, well, the entire blog. It has been a place for our "brave attempt to think out loud about theories of psychology until we get some" since day one; we've been identifying problems but, just as importantly, solutions the whole time. The theory post identified the big picture problem we see in psychology; time to lay out some solutions.

Step one is to present a map of the blog, organised thematically to guide new readers to work we've already done here. This should also help map out the gaps in the approach, so we can focus on things to do next; feel free to point us to problems we can't yet address! (And yes, we know about episodic memory and language - we're working on it.) This post is not a comprehensive summary of past work - it's a map for you to use to find what we've done so far.

To summarise: in essence, and some minor details aside, we are advocating for Chemero's (2009) radical embodied cognitive science, with the addition of some elements he was missing (network science & task specific devices). Cognition is embodied, extended and held together by the direct perception of affordances and events; the result is a complex, nonlinear dynamical system that must be analysed as such. The brain is not the sole source of our behaviour, nor is it representing the world; it clearly plays a critical role in this system, though, and we propose that we'll need the tools of network science to describe what it's actually up to (Sporns, 2010). Methodologically, we must carefully characterise the task, the resources available to solve the task (which include brain, body and environment) and the information these resources create which can sustain the formation and control of an embodied solution. This method is Bingham's (1988) task specific device approach (the main piece Chemero was missing, I think).This approach applies to all and any behaviour you want to explain, including the hard stuff like episodic memory and language.

Critically, this approach, while new (and uncommon in insisting on a role for Gibson's ecological approach) isn't just something we invented: all these elements are active parts of modern cognitive science. The only new part is bringing it all under one roof, with the goal of getting on and getting some decent normal science under our belts.

Here's what we've covered so far. If you want more details on any point, click on the links!

Cognition is embodied
The first claim we want to defend is that cognition is embodied. Embodied cognition is not the hypothesis that the contents of cognition can be affected a bit by our bodies (as implied in this study). Embodied cognition is actually the fairly radical hypothesis that the brain is not the sole resource we have available to us to solve problems. We perceive and act in very particular ways so as to generate information and solve problems non-computationally (for example, fielders catch fly balls by moving in such a way as to cancel out either the optical curvature or acceleration of the ball's motion, which happens to bring you to the right place at the right time). The bodies we move are built in very specific ways; our hands, for example, are built as if they are implementing certain computations that are required to control them. This 'morphological computation' isn't actually computation, it's more like the Watts governor (van Gelder, 1995). A great example of this idea in action is Big Dog, one of the many awesome robots built by Boston Dynamics.

Embodiment changes what 'cognition' will end up looking like. By changing the job description (e.g. what resources we have available to solve problems) we end up proposing entirely different solutions to tasks. An excellent recent book on this topic is Barrett (2011), Beyond the Brain: How body and environment shape animal and human minds. If you allow yourself bodies, behaviour and perception, then you typically don't end up needed complex computational solutions being implemented in the brain.

Cognition is extended
A logical extension to embodied cognition is the claim that cognition is extended (Clark & Chalmers, 1998). This is the claim that things in the environment literally form part of the cognitive process. This can be summarised in Clark & Chalmers' 'parity principle':

If, as we confront some task, a part of the world functions as a process which, were it done in the head, we would have no hesitation in recognizing as part of the cognitive process, then that part of the world is (so we claim) part of the cognitive process.
Clark & Chalmers, 1998, pg. 2

There is still debate about how well this idea works, mostly coming from Adams & Aizawa (2010). They believe that the hypothesis is grounded in a confusion between coupling and constitution; while we are, indeed, coupled to things in the world, they need not then constitute part of our cognition. We've had various arguments with Ken Aizawa about this (summarised here); I think the main problem with their argument is that there is no need for all the parts of a cognitive system to have 'the mark of the cognitive' if we're happy the system as a whole is cognitive. This works, I think, because of the nature of the coupling that goes on when we interact with the world: objects literally become part of us when we interact with them, and the kind of ongoing perception-action loops that support this run deep.

In order to solve a given task, then, we use a wide variety of resources; some of these are neural, but not all. Some of the resources are our bodies (our visual system is composed of mobile eyes in a mobile head on a mobile torso equipped with legs, for example), while some are objects and other people in our environments. A theory of psychology must therefore include all these resources.

The role of perception
Extended, embodied cognition requires impressive perception. Typically, perception is seen as the end point of a complex process, taking impoverished input and enriching it until it is good enough to be useful. Cognition then becomes a computational process of adding knowledge and structure to our experience. If, however, cognition is to be the solving of problems using resources distributed beyond the brain (as above) then this account isn't good enough.

We already have a theory of perception that is up to the task of providing the kind of access to the world that we need: James J Gibson's ecological approach to perception (Gibson, 1979; see the reading group posts on this book).Gibson's book begins with the environment; what is available to the perceiving organism that they might be interested in using. Starting there, rather than with the anatomy of the eye, led Gibson to propose his two key ideas: affordances, and information.

Affordances
Affordances are the opportunities for behaviour the world offers to a given organism; a handle affords grasping to a organism with a hand, for example. Technically, they are dispositions of the environment. Salt is disposed to dissolve in water, for example, but doesn't dissolve until placed in water. Affordances are dispositions supporting behaviour, but that behaviour doesn't show up until a matching organism comes by. This way of thinking of affordances makes them real properties of the world which persist in the absence of organisms (Chemero (2009) advocates treating affordances as relations (see here and here); I talked about this debate here, here and summarised it here. Long story short, I think Chemero is confusing affordances and information; the latter is relational and does every relational thing Chemero wants affordances to do, without the problems).

Some terminology (based on Turvey, 1992 & Turvey, Shaw, Reed & Mace, 1981): affordances are complex dispositional properties, composed of combinations of anchoring properties. These anchoring properties are things like the composition of surfaces - their size, shape, etc. Some organisms have complementary anchoring properties (e.g. a hand of the right size and shape) and can effect an affordance. Which properties of organisms matter is still a matter of debate: some people have proposed body scale as a key property (e.g. Warren, 1984) while other researchers feel we need something more like ability (Chemero, 2009) or effort (Proffitt, 2008). The latter is likely the right path, but is, as yet, poorly defined.

Information
Affordances are easy to define; the real question is whether there is perceptual information available to an organism for those affordances. The most detailed explanation of how affordances give rise to information is Turvey et al (1981), who lay out the concept of ecological laws to expand on Gibson's (1979) account (see this post, this post and this post on Chapter 5, and this post and this post on Chapter 6 for Gibson's description). These laws govern how the anchoring properties of affordances interact with energy such as light to create structure in light; this structure, by virtue of the law, is specific to the affordance. The laws are ecological in the sense that they have a limited scope: the law does not apply universally, but only in the kinds of niches we find ourselves. Within the scope of the law, however, ecological optics explains how affordances structure light to create information.

Because this information specifies the affordance (i.e. there is a 1:1 mapping between the optics and the world) if you detect the optical information, this is equivalent to perceiving the property of the world. Perception is therefore direct: unmediated by any internal states. As things currently stand, direct perception requires this law based, specification relationship. Whether specification is actually required is a topic of debate and we'll be getting into that soon.

The one thing that information needs to have done to it in order to be useful for the control of action is calibration. Information variables are unitless: optical information is all angular, for example, and in order to use information to act in space you must apply a metric to the measurement. This does not require internal states or processing; calibration arises from making your perceptual measurement with a ruler marked off in action relevant units (these could be body scale, or effort, etc; in other words, the units are the organism's complementary anchoring properties). The intuition here is simple; you can measure the same amount of space with a ruler marked up in, say, inches or centimetres - it's the same amount of space, but the numbers that come out are quite different. If you instead measured that space using, say, the length of your arm as a unit, you would have a number who's units tell you something directly about your ability to cross that space with your arm; this is useful for, say, reaching to grasp an object. You can directly measure all kinds of things if you have a measuring device marked up (i.e. calibrated) in the appropriate way: one example is the polar planimeter which directly measures area (Runeson, 1977).

Dynamical systems
The hypothesis that we have embodied, extended minds which rely on perception to establish the required couplings means that cognition is a complex, nonlinear dynamical system. Dynamics is the mathematical language of change over time, and provides just the right formal tools to model the kinds of systems we are. An excellent example of using dynamics to model a perception-action system is Bingham's model of coordinated rhythmic movement; this simple task is an excellent model and test bed for the ideas laid out so far.

Dynamical systems is not, in itself, a theory of behaviour; this is an error made by a lot of researchers and the data do not support them. It is, however, the right analytical tool for the job.

No mental representations
The big 'negative' thing we're going to insist on is that we're going to rule out computational mental representations as entities you can invoke in your explanations. The reason is fairly simple: there is no limit to what representations can do. Whenever you come across a problem in your explanation, say a potential lack of perceptual access to some required information, you can simply claim that the gap in your explanation is filled by a representation that has just the right size and shape. Because they can be anything you need them to do, they cease to have any explanatory power.

We've been critiquing representations since day one; Sabrina has summarised a lot of the issues here. Because representations aren't good explanations, and because when you embrace embodiment they tend to become unnecessary, we are strong advocates of the 'radical' hypothesis that we do not trade in mental representations.

Cognition does not have to be representational: the standard cry of 'what else could it be?' has been answered. van Gelder (1995) described a device, the Watts steam governor, that does a job that today would typically be solved using an algorithm and does it efficiently, reliably and stably in the face of noise and perturbations. The moral is simple; while there is an algorithm to describe the solution, it's not how the device actually works, nor would the device work well if it was. Another useful device for metaphors is the polar planimeter (Runeson, 1977); this device directly measures the 'higher order' variable area, without implementing the algorithm of 'two measurements of length combined via multiplication'. The most recent defence of a non-representational approach is Chemero's book, Radical Embodied Cognitive Science. So while you may not agree with the idea, a non-representational cognitive science is, at least, a viable option, and we believe one justified by taking embodiment seriously.

(I would love to see an inventory of all the representations that have ever been invoked in cognitive science; I would guess that it is a tangled and incoherent mess of things designed to just fix that one problem.)

What about the brain?

What is the brain doing, if not representing? Are ecological psychologists really committed to the idea that brains don't matter? No, of course we aren't; we only look like we are because instead of plunging off the deep end with rampant speculation about what the brain is up to we've spent our time working out what it has to work with. First things first!

The brain is clearly important, it's just not representing anything; rather the brain is in a constant state of change in response to it's environment. I'm inclined right now to treat it as the fast responding resource which coordinates the assembly of task specific devices, as well as a system that can implement embodied solutions to computational problems.

The most promising approach to neuroscience I've seen recently is that described by Olaf Sporns' book Networks of the Brain (Sporns, 2010). In this book, Sporns describes how the mathematics of networks are being applied to neuroscience datasets to uncover structure extended over both space and time within the endless modelling and remodelling of neural connections. This seems to me to be the right toolset for neuroscience; combined with our radical, embodied cognitive science it could be a powerful approach, and we're waiting to hear about funding for a project to set this idea in motion.

Summary
There's a lot of work to do. But these are our core theoretical commitments - when we try to explain our data, we must characterise the task resources (which can include the brain, body and the environment) as well as the information supporting the coupling of these resources into task specific devices which solve the current task. The tools exist, and there are plenty of problems to study. Psychologists will need to get a little better at physics, biology, and maths, and we'll need help from experts in these fields. But I truly believe that taking this strong theoretical stance will allow psychology to apply itself to a coordinated programme of research that, right or wrong, will produce a wealth of data and drive our understanding for. After all, that's what a theory is for.

References
Adams, F., & Aizawa, K. (2010). The Bounds of Cognition. Wiley:Blackwell. Amazon.co.uk

Barrett, L. (2011). Beyond the Brain: How body and environment shape animal and human minds. New Jersey: Princeton University Press. Amazon.co.uk

Chemero, A. (2009). Radical Embodied Cognitive Science. Cambridge, MA: MIT Press. Amazon.co.uk

Clark, A., & Chalmers, D. (1998). The extended mind. Analysis, 58 (1), 7-19 DOI: 10.1111/1467-8284.00096 Download

Gibson, J.J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin. Amazon.co.uk

Proffitt, D. R. (2008). An action-specific approach to spatial perception. In R. L. Klatzky, M. Behrmann, & B. MacWhinney (Eds.), Embodiment,ego-space, and action (pp. 179–202). Mahwah, NJ: Erlbaum.

Runeson, S. (1977). On the possibility of "smart" perceptual mechanisms Scandinavian Journal of Psychology, 18 (1), 172-179 DOI: 10.1111/j.1467-9450.1977.tb00274.x

Sporns, O. (2010) Networks of the Brain. Cambridge, MA: MIT Press. Amazon.co.uk

Turvey, M. (1992). Affordances and Prospective Control: An outline of the ontology. Ecological Psychology, 4 (3), 173-187 DOI: 10.1207/s15326969eco0403_3

Turvey, M., Shaw, R., Reed, E., & Mace, W. (1981). Ecological laws of perceiving and acting: In reply to Fodor and Pylyshyn (1981) Cognition, 9 (3), 237-304 DOI: 10.1016/0010-0277(81)90002-0

van Gelder, T (1995). What might cognition be, if not computation? The Journal of Philosophy, 92 (7), 345-381 Download

Warren, W. (1984). Perceiving affordances: Visual guidance of stair climbing. Journal of Experimental Psychology: Human Perception and Performance, 10 (5), 683-703 DOI: 10.1037/0096-1523.10.5.683 Download

This is not the paternity leave policy you were looking for

2011-12-06T13:31:00.000Z

So, this is a bit off the scientific/psychological beaten path, but since we get a good amount of traffic from people in the UK it seemed worth mentioning...

Image via Hammonton Photography

You might be aware that the UK government recently did something awesome by giving employed fathers the right to take up to 26 weeks off work during the first year of their child's life. This was a massive step forward from the previous Ordinary Paternity Leave policy, which gave dads a whopping 2 weeks off work. In terms of government mandated pay, the scheme is pretty close to the maternity leave policy; dads receive the lesser of £128.73 a week, or 90% of average weekly pay. One difference, is that fathers can only begin their leave after the mother takes 20 weeks of her leave.

Many employers (universities among them) offer additional pay for maternity leave to employees within certain salary band (which is great- thanks!). But, (here's where things aren't so great), some employers don't feel that this extra pay should extend equally to paternity leave (warning - I don't have national data on this; my conclusion is based on the policies I've seen from a handful of institutions). For example, a maternity leave plan might offer extra pay to mothers until 26 or 31 weeks, while fathers who start their leave at 21 weeks would receive no extra pay (beyond the statutory rate mentioned above). This means that, in practice, maternity leave policies are more financially generous than paternity leave policies. Sharing leave between parents is often more financially detrimental to a family than having the mother take the entire leave.

I could better understand employers' positions if the new law allowed both parents to take leave at the same time (because this would cost the employer twice as much money over that period). But, fathers can only take leave if the mother has officially returned to work (she is also not allowed to take annual leave to extend this period). This means that the total amount of time that parents can take leave from work remains the same. The only difference is whether the mother or the father is the one at home.

Obviously, equating paternity and maternity pay would hit some institutions harder than others. Employers with a predominantly male work force would find themselves paying out substantial sums in leave pay for the first time. In contrast, business that employ predominantly women would likely save money as some mothers return to work weeks or months earlier than they would have done without the new law. Also, to the extent that the gender pay gap remains, employers who base additional leave pay on salary (half salary plus statutory maternity pay is a common option) would be paying out more money to cover higher paid paternity leave.

I can see why employers facing these sorts of situations would not want to shell out extra money on paternity leave. But, that doesn't mean that having different policies is okay. It's not. Creating financial incentives for mothers to take more leave than fathers is unfair for two reasons:

1) It is unfair to fathers who are less able to equally participate in early parenting. Cultural expectations of fathers are slowly shifting and this is reflected in the new law. But, individual employer implementation of the law reinforces the idea that "it's great if a dad wants to help out with child care" rather than "it is a father's responsibility to share equally in child care."

2) It is unfair to mothers who will continue to take the hit in terms of career advancement and employment opportunities. Check out these popular (and depressing) news reports on the subject.

The government reports that it will update the paternity leave policy by 2015; let's hope that new law includes a mandate for employers to offer equal benefits to mothers and fathers.

Do you know what the paternity leave policy is where you work?

*I should mention that I am an American living in the UK. As such, I am grateful every single day to live in a country that has ANY sort of comprehensive paid maternity/paternity leave*

How Universal Is The Mind?

2011-11-29T13:11:00.000Z

If someone asked you to describe the psychological aspects of personhood, what would you say? Chances are, you'd describe things like thought, memory, problem-solving, reasoning, maybe emotion. In other words, you probably list the major headings of a cognitive psychology text-book. In cognitive psychology, we seem to take it for granted that these are, objectively, the primary components of "the mind" (even if you reject a mind/body dualism, you probably accept some notion that there are psychological processes similar to the ones listed above). I've posted previously about whether the distinction between cognitive and non-cognitive even makes sense. But, here, I want to think about the universality of the "mind" concept and its relationship to the modern view of cognition.

In fact, this conception of the mind is heavily influenced by a particular (Western) cultural background. Other cultures assign different characteristics and abilities to the psychological aspects of personhood. Wierzbicka (2005) delves into this problem in detail. She argues that speakers of a particular language make assumptions about what must be universal based on their own ability to imagine doing without a certain concept. Important cross-cultural differences in meaning become lost in translation. For instance, Piaget’s “The moral judgment of the child” was translated to English by substituting the French “juste” with the English “fair.” So, English readers think they are reading about the development of fairness in children, when this was not the author’s intention.

Translation is a deep problem, but it is often ignored in psychology. Generalisations about cognition must be made in some language, but, language is specific to particular cultures. Our choice of language, then, inevitably will bias how we talk about cognition across cultures.

“If we uncritically formulate some hypothetical universals in one particular natural language, for example, English, we run the risk of distorting them by imposing on them the perspective embedded in that particular language; and the same applies to our description of cultural differences” (Wierzbicka, 2005; p. 257).

So, back to the mind and our current view of cognition. Cross-linguistic research shows that, generally speaking, every culture has a folk model of a person consisting of visible and invisible (psychological) aspects (Wierzbicka, 2005). While there is agreement that the visible part of the person refers to the body, there is considerable variation in how different cultures think about the invisible (psychological) part. In the West, and, specifically, in the English-speaking West, the psychological aspect of personhood is closely related to the concept of "the mind" and the modern view of cognition.But, how universal is this conception? How do speakers of other languages think about the psychological aspect of personhood?

In Korean, the concept "maum" replaces the concept "mind". "Maum" has no English counterpart, but is sometimes translated as "heart". Apparently, "maum" is the "seat of emotions, motivation, and "goodness" in a human being" (Wierzbicka, 2005; p. 271). Intellect and cognitive functions are captured by the Korean "meli" (head). But, "maum" is clearly the counterpart to "mind" in terms of the psychological part of the person. For example, there are tons of Korean books about "maum" and body in the same way that there are English texts on "mind" and body.

The Japanese have yet another concept for the invisible part of the person - "kokoro"."Kokoro" is a "seat of emotion, and also, a source of culturally valued attention to, and empathy with, other people" (Wierzbicka, 2005; p. 272). To illustrate the contrast between "kokoro" and "mind", Wierzbicka gives the following example: A Japanese television programme proclaims, "The 21st century should be the age of kokoro. Let's make a point of meeting with other people" (Hasada, 2000: 110). If an English speaker declared the 21st century to be "the age of the mind" then "meeting with other people" probably would not be a priority - thinking and knowing would be. In contrast to the Korean "maum", "kokoro" is not associated with will and motivation ("hara" meaning belly serves this purpose in Japanese). But, "hara" is not associated with the psychological component of the body, the way "kokoro" is. In other words, "maum" is all about motivation and "kokoro" is all about feelings and "mind" is all about thinking.

Interestingly, Russia, which kind of sits between East and West uses "dusa" as the counterpart to the psychological part of the person. "Dusa" is often translated as "soul", but also sometimes as "heart" or "mind." "Dusa" is associated with feelings, morality, and spirituality. The "dusa" is responsible for the ability to connect with other people. This meaning seems to lie somewhat more with the Eastern conception than with the highly cognitive concept of "mind."

In a larger sense, the fact that there seems to be a universal belief that people consist of visible and invisible aspects explains much of the appeal of cognitive psychology over behaviourism. Cognitive psychology allows us to invoke invisible, internal states as causes of behaviour, which fits nicely with the broad, cultural assumption that the mind causes us to act in certain ways.

To the extent that you agree that the modern conception of "cognition" is strongly related to the Western, English-speaking view of "the mind", it is worth asking what cognitive psychology would look like if it had developed in Japan or Russia. Would text-books have chapter headings on the ability to connect with other people (kokoro) or feelings or morality (dusa) instead of on decision-making and memory? This possibility highlights the potential arbitrariness of how we've carved up the psychological realm - what we take for objective reality is revealed to be shaped by culture and language.

Hasada, Rie (2000). An Exploratory Study of Expression of Emotions in Japanese: Towards a Semantic
Interpretation. Ph.D. thesis, Australian National University..

Wierzbicka, A. (2005). Empirical universals of language as a basis for the study of other human universals and as a tool for exploring cross-cultural differences. Ethos, 33(2), 256-291.

Theory, and Why It's Time Psychology Got One

2011-11-10T14:20:00.000Z

Psychology has a problem. We have no core theory to guide our research; no analogue to the theories of evolution or relativity. When particle physicists recently found that some neutrinos had apparently travelled faster than light, it never actually occurred to them that this is what had happened. On the basis of the extraordinarily well supported theory of relativity, everyone went 'huh, that's weird - I wonder what we did wrong?', and proceeded to use that theory to generate hypotheses they could then test. It would take a lot of fast neutrinos to disprove relativity.

Psychology, though, when faced with an empirical result that violates the laws of physics, can't find any principled reason to reject the result and instead spends a lot of time squabbling about whether Bem's result might possibly be true because 'quantum'. Worse, when people do replicate the experiment and fail to support the original result, they can't get their 'null result' published. It's a bit embarrassing, really.

One of the problems of having no core theory is that you can't simply rule things out as options. Psychologists almost all consider this a strength: we can pick and choose from a variety of mechanisms which enables us to cope with our messy and erratic subject matter. Can't imagine how perception can explain a result? Just hypothesise a mental representation to fill the gap. After all, no single theory is going to account for the opportunistic and idiosyncratic behaviour of people, so why limit ourselves? We tried that with behaviourism, and it got us nowhere. Let's stay flexible.

The problem with this approach is that psychology has gotten lazy; when you can't come up with a simple solution to your complex problem, you suggest a complex solution that fills all those pesky gaps, and never notice the gaps were a bit weird to begin with (Costall, 1984). Representations can solve everything if they contain the solution, and thus they explain nothing at all. An example of this comes from the prehension literature: when reaching, people accelerate their hands to a peak speed that occurs ~70% of the way through the reach. To explain this stable timing, researchers like Jeannerod postulated motor programmes that simply instructed the system to reach this way; they just took the structure of the reach and placed it in a representation. This explains nothing: the structure in the reach is caused by the structure in the representation, but what caused the structure in the representation? This question got more complicated as it became clear that the details of the temporal structure of prehension emerge in real time as a function of the task; suddenly you needed a different programme for every different situation! The answer actually requires a careful analysis of the affordances of the task (e.g. Mon-Williams & Bingham, 2011). This, however, is hard.

This laziness and lack of discipline leads people to try and combine arguments and mechanisms that simply aren't compatible with one another. Do you like embodiment, but can't see how cognition can occur without representation? Let's have embodied representations! As Louise Barrett's book makes clear, however, this type of approach does nothing but make you miss the actual embodied solution. Her account of the wonderfully sophisticated behaviour of the Portia spider talks about how people first explained their ability to navigate via detours in terms of 'insight'; they sit and scan their environment and then suddenly move off in the right direction as if they had planned their route. The form of the scanning behaviour itself, however, is critical, and careful experiments that break the information they are looking for have shown that Portia scans and moves according to simple rules that lead to the right solution (like the 'linear optical trajectory' solution to the outfielder problem). Tony Chemero also lays out the fact that embodiment and representations come from irreconcilable schools of thought quite clearly; so not only is it bad science, it's incoherent to begin with. We simply can't keep picking and choosing the easiest explanations; it's time to invest the time and effort required to uncover what's actually going on. It's time to spend some time working within the confines of a real theory.

Theory in Science
As I try to teach my students, the role of theory in science is to provide structure to your data. A good theory rules explanations in and out, and if it rules out the wrong explanation that will become clear over time as you pursue your theory guided research. Good science means a) restricting your explanatory mechanisms to include things your theory allows, and b) keeping an eye on how well that's working out for you, while c) allowing yourself to rest on your well supported theory to resist breaking the rules as long as you can. When creationists say 'we can't see how the chambered eye could possibly have evolved, it's far too complex', biologists are entitled to say 'well, we have evidence that lots and lots of other things have evolved, let's see if we can figure out how the eye did it, and in the meantime, we're going to operate on the assumption that it did evolve until we have strong evidence to the contrary'. No-one except creationists complain about this; it's perfectly healthy science.

In psychology, when Bem publishes a paper that rests on the assumption of information travelling backwards in time, I get told off for simply ruling out his explanation for his data on the basis that a) it's physically impossible and b) 'quantum' isn't a get out of jail free card for (a). According to my anonymous debater, it's bad science to rule things out that don't fit my theories of how things work, because I might be wrong. Yes, I might be wrong: but good hypothesis driven science will eventually reveal this if it is, indeed, the case, and that's what the scientific method is for. I am actually allowed to rest on the well supported theory of relativity that describes why Bem's explanation is is impossible and simply reject the explanation out of hand. This doesn't make me close minded, this makes me a scientist. If neutrino research actually manages to break relativity, then I will reconsider; in the meantime, Bem is simply cheating and I'm calling him out for it.

Ecological Psychology
I think the closest psychology has to a decent actual theory of behaviour is Gibson's ecological psychology; this is Chemero's bet too. It proposes very specific hypotheses to explain behaviour; these hypotheses contain suggested mechanisms (specifically, information and affordances) to support behaviour, and suggests ways to empirically test these hypotheses. These tests have been very successful (e.g. my work in coordinated rhythmic movement, and the success of the three key predictions of the perception-action model, not to mention affordance research coming out of everywhere).

Because of these successes, I am able to use this theory to generate predictions about other behaviours like catching a fly ball). There are two ways to achieve a goal related to the future state of things: prediction, and prospective control. The former entails taking the current conditions and using these to predict future conditions, then acting on the basis of that prediction. The latter entails coupling your behaviour to specific aspects of current conditions, and letting the future solution emerge as you engage in perceptually controlled behaviour. The ecological approach rules out the former as an option, and goes looking for evidence of the latter. My work based on this theory will either work or it won't; but at least I'll be able to tell the difference, if my empirical work is theoretically constrained.

The Cost of No Theory
Psychologists hate ruling things out - they just love leaving the door open to 'both solutions are probably used', because they have no particular reason to rule anything out. But this has a huge cost: psychology becomes a mere collection of empirical results, with nothing tying them together. Results from the different disciplines can't inform each other, because they aren't testing the same things. This is currently most clear to me in the gulf between embodied cognition and cognitive neuroscience. Neuroscience finds computation and representation only because it's looking for computation and representation: no-one is asking non-computational, non-representational questions or interpreting data in that light, so there's nothing in the neuroscience literature that informs my science.

This fragmentation means psychology is doing nothing but running in empirical circles: there's nothing resembling progress. All you get are individuals with their own collection of hunches running their own experiments on their own little experimental phenomena. Psychology needs to pick a side, suck it up and get on with some normal science for a change. Taking embodiment seriously is the first step. Taking perception seriously is the second. Using dynamical systems to describe these things is not a bad idea either. Our empirical successes are mounting, whenever anyone bothers to go looking - Gibson and embodied cognition seem a healthy place to start, because representational approaches simply don't rule anything out and thus can't help.

And the beauty of a period of serious normal science is that if we invest some serious time pushing the theory, looking for cracks, and resisting the temptation to jump ship at the first sign of trouble, we will end up in a better place no matter how it pans out. If the theory breaks, it will have been broken honestly, and for good reasons. If the theory holds up, we will have achieved a lot of progress and begun to act like a real science for a change. This is why the scientific method is awesome: because it works, bitches.

References
Costall, A. P. (1984). Are theories of perception necessary? A review of Gibson's 'The Ecological Approach to Visual Perception'. Journal of the Experimental Analysis of Behavior, 41(1), 109-115. Download

Mon-Williams, M. & Bingham, G.P. (2011). Discovering affordances that determine the spatial structure of reach-to-grasp movements. Experimental Brain Research, 211(1), 145-160. Download

Embodied cognition is not what you think it is

2011-11-06T12:04:00.002Z

The most exciting thing in cognitive science right now is the theory that cognition is embodied. It is, in fact one of the things interested lay people know about cognitive science, thanks in part to a lot of high profile experiments that claim to show how cognition can be influenced and biased by states of the body, or that cognitive states can affect states of the body in ways that suggest abstract metaphors and concepts are grounded in the behaviour of the body. A recent blog post at Scientific American covers this ground quite nicely (although the large picture of Noam "not an embodied cognitive scientist" Chomsky hints that someone didn't actually read the piece).

The problem, however, is that this is not really what embodied cognition is about. Embodiment is not the weak claim that you can see small effects of the behaviour of the body in our mental representations of the world. Embodiment is the radical hypothesis that the brain is not the sole resource we have available to us to solve problems. Our bodies, and the meaning-filled perception of the world they allow, do much of the work required to achieve our goals, and this simple fact changes utterly what our theories of 'cognition' will look like.

Disembodied cognition
The original computational depiction of cognition in the 1960s treats us as general problem solving devices. Similar tasks (e.g. reaching with your left vs. your right hand) are achieved by applying the same underlying general motor programme to the different effector. The body was simply the output system attached to the cognitive processing system; it was cognition that was responsible for all the characteristics of what the body gets up to. More recently, people have begun to realise that the types of bodies contribute non-trivially to the form of our various behaviours, and the idea of 'embodied cognition' began to emerge.

Then, like everything else in psychology, it immediately fractured into approximately 5 different versions. Most of these are wrong, or, at the very least, nowhere near as new as they like to pretend.

What embodied cognition isn't
The most commonly discussed embodied cognition hypothesis that the contents of our mental representations can be affected by states of the body (this is what that Scientific American article considers embodied cognition). For example, the way we mentally represent abstract concepts seem to be influenced by physical metaphors - we might conceive of importance as weight, and thus the weight of things might affect how important we think they are (Jostmann et al, 2009). Another way to demonstrate this is to find changes in states of the body as a function of changes in our mental states; the suggestion is that if the mental state is grounded in a sensorimotor experience, activating the former might also prime or activate the latter. An example of this is the 'moving through time' paper (Miles et al, 2010), in which people thinking about the future tended to sway forwards (on average) and those thinking about the past tended to sway (on average) backwards. (I've critiqued this paper fairly comprehensively here; funny how these tend to show up in Psychological Science!).

Besides the tendency to over interpret tiny effects and generally ignore what the perception-action system might actually be up to, the core problem with these versions of embodied cognition is this - there's nothing new or interesting here. All these papers assume that the contents of cognition are, roughly, what we've thought it was since the 60s - mental representations containing abstract, generalised and computational programmes that enable us to process impoverished sensory data into a perception of the world good enough to support skilled action. The only difference is that these researchers suggest these programmes can be influenced by some of the things we get up to. The strongest claim you will ever hear is that these programmes can finally be grounded, i.e. there is a reason (embodiment) that they have the contents they have.

Why is this such a problem? The hypotheses about these mental states, and what kind of content they have, came from the specific work they have to do in order to enable behaviour. For example, a representation may contain a prior probability distribution that suggests a given pattern of sensory information is probably caused by a particular state of affairs in the world, or it may contain a mapping between an event in the world and the motor system's reponse. This flavour of embodied cognition simply claims that representations still have to do the same kind of work, but that now some of their content might reflect something about the state of the body. This misses the fact that embodiment is actually a much more radical hypothesis, and that it's first implication is that cognition now has a very different job description.

The radical implications of taking embodiment even a little bit seriously
What I think the real hypothesis of embodied cognition is that the type of perception and action systems we have radically alters what 'cognition' needs to look like. My favourite current example is the outfielder problem (how a baseball outfielder is able to catch a fly ball; McBeath et al, 1995), and the contrast between the straight-forward cognitive, predictive strategy versus the perception-based prospective control solutions. The contents of cognition for the predictive strategy includes a) an estimate of the initial conditions of the ball's flight, derived from perception, b) an internal model of projectile motion which can take those initial conditions as input and c) a mapping from that model to the motor control system to allow you to move in the right direction.

The embodied solutions, for which there is clear evidence, utterly change this content. You no longer require an internal model of projectile motion; you simply need the ability to visually perceive the motion of the ball and the experience to move so as to produce a particular pattern to the optical motion of the ball. So it would make no sense to go looking for effects of the body on your internal model of projectile motion, because, when you take embodiment seriously, you cease to think there will be one.

Other examples abound in the literature; some highlights

rat pup huddling critically depends on the shape of rat pups
Barrett (2011) describes the implausible nature of the Portia spider, who accomplishes great feats of complex navigation that emerge from the spatial arrangement of their eyes and the scanning behaviour afforded by that arrangement.
the size-weight illusion doesn't come from people mentally accounting for object size when perceiving weight, it's the result of people correctly perceiving the throwability of objects
keas and crows are excellent problem solvers, but do so in different ways that critically depend on the affordances of their bodies
embodied robots require little in the way of computational power, but are capable of all kinds of flexible behaviour. This behaviour often changes with simple changes to their body structure; Pfeifer & Scheier (1999; see also Pfeifer & Bongard, 2007) described their 'Swiss robots' who clean up their environment because of the relationship between the angle between their light sensors and the simple rule governing their behaviour; change the angle and not the rule, and their behaviour is qualitatively different.

Any good book on embodied cognition will include a review of empirical results like this, and many of the more recent ones (e.g. Barrett, 2011; Chemero, 2009) also tackle examples of embodied solutions to more traditional cognitive tasks involving memory, or abstract problem solving, rather than just the perception-action ones I typically talk about. My bias here is partly because it's the literature I know, but also because my discipline is where a lot of this work has been done. There are, however, an increasing number of researcher in other fields beginning to take this approach seriously.

Pick a side, people
But surely there can be more than one approach to embodied cognition, and my version is no more or less right, just different. Right? Wrong. First of all, this (very typical) argument is one of the main weaknesses of modern psychology - it's time to pick a theoretical side, already, because these things aren't just different, they are often directly contradictory. I'll go into that in my next post.

I don't claim I'm right and they're wrong because I think the specific implementations of embodied cognition I talk about are 'true'. I do so because the underlying theory that produced these implementations is more true to the implications of embodiment. All embodied cognitive scientists claim that the type of bodies we have affect what the final form of cognition will be, but only the second group is taking the implications seriously. The first group have simply taken disembodied cognition and allowed the body to quantitatively affect those states (by a main effect, or, if you're feeling snazzy, an interaction!). The second group have noticed that embodiment actually qualitatively changes what cognition will be; the difference is one of kind. And this is why I say the first group are wrong: they claim to take embodiment seriously, but their research remains business as usual, with a couple of embodied bells and whistles.

A note on representations
The hypothesis of embodied cognition is not necessarily anti-representational; much of it, even the good stuff, is not committed to losing representations. Andy Clark famously reserves some room in his theorising for 'representationally hungry' problems, even though he's happy for embodied solutions to show up a long way into cognitive territory. Barrett's book also doesn't deny representations, although she is quite close to realising she doesn't need them and notes she read Chemero's book too late to work it in detail into hers. But I actually agree with Chemero, that trying to do representational embodied cognitive science is a bit of a category error. You simply end up with an utterly different job description for cognition, and also, I think, for the brain.

One reason psychologists persist with representations is that they simply can't see what else cognition could possibly be, if not representational and computational. There's simply no excuse for this any more, though: Chemero's book is an excellent and clear exposition of a viable alternative framework, one which was first crystallised by van Gelder's analogy of the Watts steam governor (laid out in detail by Sabrina here). As I tend to (somewhat rudely) reply to complaints of 'I cannot see how this could possibly explain [insert what this person studies here]', your failure of imagination should not be mistaken for a problem for my theory, and it's past time the field started taking the implications of embodiment seriously.

Summary
Embodied cognition is not about letting the body nudge the contents of cognition - it's about treating the body (and the environment, through our embodied perceptual explorations of that environment) as critical elements in a broader cognitive system, and the shape of that system is different from anything we've thought about before in cognitive science. Embracing embodiment is a radical move, but as the evidence continues to roll in now we know what kinds of questions to ask, it's clearly the future of cognitive science.

References
Barrett, L (2011). Beyond the Brain: How Body and Environment Shape Animal and Human Minds. Princeton University Press; New Jersey.

Jostmann, N., Lakens, D., & Schubert, T. (2009). Weight as an Embodiment of Importance Psychological Science, 20 (9), 1169-1174 DOI: 10.1111/j.1467-9280.2009.02426.x Download

McBeath, M.K., Shaffer, D. M., & Kaiser, M.K. (1995). How baseball outfielders determine where to run to catch fly balls. Science, 268, 569-73. Download

Miles, L., Nind, L., Macrae, C. (2010). Moving Through Time Psychological Science, 21 (2), 222-223 DOI: 10.1177/0956797609359333 Download

Pfeifer, R., and Bongard, J. (2007). How the Body Shapes the Way We Think. MIT Press, Cambridge, Mass.

Pfeifer, R., and Scheier, C. 1999. Understanding Intelligence. MIT Press, Cambridge, Mass.

Robots, Representation, & Dynamical Systems

2011-11-04T13:00:00.000Z

When cognitive science tries to explain a given behaviour, it typically looks in one of two places for it's explanation. Some people go looking in the brain for the representation that encodes the solution to the task; these people typically treat the brain as the source of the observed structure in behaviour. Some of us, however, go looking in perception for the necessary access to the properties of the world that enable us to couple our resources to those in the environment. We consider the origin of behaviour to be the dynamical system formed by this coupling; the system provides a set of constraints and behaviour emerges as the constrained system works over time.

It's sometimes a little hard to tease these suggestions apart: after all, they typically both predict that we succeed at the task at hand. When studying people, the best way to try and separate these two suggestions out is to examine how we succeed. For instance, in catching a fly ball, the brain-based prediction solution says we will run in a direct line to where we think the ball will land; the perception-based coupling solution suggests we will run along curved paths as we attempt to move so as to produce the information required. In this case, data supports the latter hypothesis, but it's not always that easy.

Sometimes, you've just got to start from scratch and build yourself a robot.

Robotics: where the cognitive rubber meets the road
A lot of work in cognitive science involves robots of one kind of another. Robots are useful, because you can build them according to a range of design principles, and you will know exactly what the system does and does not have available to it. For example, Barrett (2011) reviews some work by Barbara Webb, who has built robot crickets that contain very simple nervous systems and ears placed in particular configurations. These robots produce all the mate selection behaviour seen in real crickets, and are a strong proof of the claim that real crickets also behave in this embodied fashion because Webb knows exactly what her robots can and can't do.

My two favourite perception-action robots are Asimo and Big Dog. They are each exemplars of the different approaches to explaining behaviour, and what they can and can't do is highly informative because we know exactly how each was built.

The Information Processing Approach: Asimo

Run, children, run!

Asimo is a robot made by Honda, to showcase their technological skills. He literally implements an information-processing, representational style of control: his on-board computers perform sophisticated image processing on 2D snapshots of the world, and every one of his movements is controlled explicitly by programmes he has been given that transform this visual input into motor commands. With the right programme, he can walk, run, dance and climb stairs. He is quite slow; it takes him time to select the correct action, for instance. However, Honda suggest that this is merely a state-of-the-art problem: as computer hardware improves, Asimo's performance will simply get better and better.

However, even with the correct program, configured for the current task, Asimo can fail in informative ways:

Asimo misses the stair by a tiny amount, and utterly fails to catch himself. I show this video in class, and ask students whether they've ever missed a step by such a small amount - most have. I then ask what happened, and only a couple have ever had a serious fall as a result. Critical failures are, of course, possible - but they aren't common, which is lucky because they have serious consequences.

Four legs good.

The Dynamical Systems Approach: Big Dog
Marc Raibert and the team at the MIT Leg Lab were interested in building stable robots with legs who could walk and maintain balance. Their approach was grounded in dynamical systems, and instead of complex computers solving algorithms to maintain balance, they simply built robots with springy legs and joints that mimic the kind of dynamical system found in bipedal locomotion, specifically the dynamic of a bouncing ball. The Leg Lab built numerous examples of legged robots in which the work of producing gait patterns, etc emerged from the way the robots were built.

Raibert left MIT and formed Boston Dynamics to develop these robots for commercial and military applications. The company's star is Big Dog, a quadruped robot who can traverse unstable terrain and carry heavy loads.

Big Dog has very little brain; he has many sensors tracking the state of his various components (power, etc) but nothing dictating the specific form of his motion. He goes where he's told and his legs do what they need to do to get him there.

The key comparison with Asimo comes 30s into the above video; Big Dog is given a huge kick from the side, and simply catches himself in a remarkably biological-looking manner (watch his legs splay out to catch himself). Big Dog does not have a programme that enables him to cope with this perturbation. He is simply built in a way that enables him to cope with the perturbation; his response capabilities are literally embodied, and therefore fast. In contrast, Asimo falls because he a) doesn't have a programme that tells him how to respond to this situation and b) because even if he did have such a programme, it would take him too long to find it and implement it.

Summary
Big Dog's behaviour emerges from his embodiment; Asimo's is imposed by his programming. We act a lot more like Big Dog than Asimo, and the fact that you can get this kind of sophisticated behaviour without any explicit planning or representational control is an existence proof of a new way of thinking about behaviour.

Robotics is to psychology as engineering is to physics: it's where your elegant theory either builds a robot that works or it doesn't. Dynamical robots can be made to produce all kinds of complex behaviour by simply being built in a specific way that relates to it's environment in a specific way; they are task specific devices, and smart ones at that. Information processing robots are slow, unstable, and occasionally beat humans at chess (something we aren't all that good at anyway). Dynamical robots can typically only do their one thing very well; but then, they don't have nervous systems which support the quick assembly and disassembly of resources into different devices. If you implement this soft assembly for them (by, perhaps, moving some of their sensors or altering the dynamics of their limbs) then different behaviour emerges; fixing this one lack of flexibility is, therefore, simply just a matter of making a robot that can do that for itself.

Further Reading
Barrett, L (2011). Beyond the Brain: How Body and Environment Shape Animal and Human Minds. Princeton University Press; New Jersey.

Pfeifer, R., and Bongard, J. (2007). How the Body Shapes the Way We Think. MIT Press, Cambridge, Mass.

Pfeifer, R., and Scheier, C. 1999. Understanding Intelligence. MIT Press, Cambridge, Mass.

The Affordances of Everyday Things

2011-10-29T21:34:00.000+01:00

I ramble on about affordances quite a bit; a big chunk of those posts was dedicated to figuring out whether they are relations or dispositions (answer: they're dispositions), and then there was all that arguing with Ken Aizawa about whether they are even anything at all (answer: yes, they are).

The thing I've been a little worried about is that people might see all this and come away thinking, "Why should I care one way or another?". I wanted to spend a post reminding everyone why all this talk about affordances matters, and the reason is simply this: affordances are important because they influence our behaviour, for good and ill, all day, every day. I'd like to illustrate this with a story, and a few photos I've been accumulating recently.


The McDonald's on O'Connell St in Dublin, with doors open for a change

Many years ago, I had come to Dublin to visit a good friend of mine, and we'd arranged to meet in town at this McDonald's; the bus from the airport stops there, so it was handy. I arrived a little early, and went inside to sit down. The doors that day were both closed, and presented me with vertical bars for handles. I then went through a routine to get in, and spent the next 20 minutes or so watching everyone else who entered the store do exactly the same routine, which went like this:

Reach out with right hand to pull on the right door's handle. The door doesn't move.
Think, 'you must have to push', and so push the door. The door doesn't move. This door is locked.
Check that the store is open. It is.
Reach out with left hand to pull on the left door's handle. The door doesn't move.
Swear.
Try pushing the left door, not expecting anything to happen. Door moves, letting you stumble into the store in an extremely bad mood.

I find this sequence very revealing. People tried to most obvious thing first: pulling the door closest their favoured right hands (I blame me doing this on living in a right handed world). Pulling was an attempt to effect an affordance; going right reflects a bias in design. They then pushed; people are used to dealing with stupid doors and know they often need pushing. They then went to the other door and didn't try the push, the last thing they had done; instead, they immediately tried to effect the affordance and pull again. Finally they push and get it right, but only because there was no other combination available.

Donald Norman and "The Psychology of Everyday Things"
If you've ever read Donald Norman's classic book, The Psychology of Everyday Things, you will be very aware of the existence of doors with handles that you have to push. Norman talks explicitly about the affordances of objects, and how the things we interact with all present a way of interacting with them to us. (Norman has read his Gibson and taken the concept from there, although he is very up front about considering them to be cognitive, mental entities. He knows this isn't what Gibson meant, though, he simply disagrees; I can respect this, because he's clearly paid attention.) The book is just a wonderful catalogue of objects that have been poorly designed and solutions based in affordances for fixing them; if you've never read it, you should because it's a classic and extremely readable, and you'll never look at doors and taps the same way again.

These failures amaze me. Take that McDonald's. They had gone to some expense to present me with two doors, each with handles just begging to be grasped and pulled by a human hand. They had then gone to the trouble of making the doors work by pushing, and, for good measure, made one door into a wall by locking it. None of these actual facts about the doors were perceptually available. Quite the contrary: the affordances of the doors were the direct opposite, for pulling, and I acted accordingly (as did at least 20 people after me).

I've always thought that McDonald's (and everyone else responsible for the terrible doors in the world) should care more about this. Handles cost more than no handles; two doors cost more than one door; and making people feel like idiots because they can't operate one of the simplest devices in the world on their way to a burger they might feel a bit guilty about eating anyway is just terrible PR. And, as Norman ably describes, it just doesn't have to be this way. You can make doors whose function is clearly specified; you can make taps you only need one attempt to get water from, and oven element controls that you get right first time. These 'gulfs of expectation and execution' are not inevitable, and good design is about making these gulfs as small as possible by bringing the function of an object to where it can be perceived by the user.

I've uploaded and captioned some photos I've taken recently to a Picasa album, which you can view in full here (if you have pictures of your own, send them to me, I collect them to use in lecture!). Once you start seeing these things, you'll never go back; and the thing that you will notice the most is just how common poor design is. The vast majority of doors you will interact will suck at being doors, and everywhere you go, taps will require you to do something different to get water from them.

I highlight all this to make my point: affordances are important because they influence our behaviour, for good and ill, all day, every day, and nothing makes that more obvious than the bizarre failures of design we interact with all the time.

This label means they did it wrong

The 'undesigned' world tells fewer lies, fortunately, because their affordances are typically grounded in anchoring properties that lead to reliable, even lawful consequences for what the object looks like. Of course, it's not perfect: the problem of perceiving the affordances for locomotion on ice reminded me quite forcefully that not all anchoring properties have consequences are specified in information, and we only perceive what there is information for. In the case of ice, the critical property (friction) doesn't exist until my foot contacts the icy surface, so there is no information ahead of time, only once I'm on the ice. And this is essentially Donald's lesson too: the key properties of the object always exist, but when poor design or no information hides these properties from perception, disaster ensues.

Donald has a simple rule for designers: "If you have to add verbal instructions, you designed it wrong"; next time you're out, count how many simple things like doors have verbal instructions, and you'll wonder how we ever let it get this way.

References
Norman, D. (1988). The Psychology of Everyday Things. Basic Books. Amazon.co.uk

Review: Louise Barrett's "Beyond the Brain"

2011-10-25T17:32:00.002+01:00

There are a lot of books about embodied cognition. Like psychology itself, the idea that cognition might not all be in the head has fractured off into about 5 distinct flavours, most of them quite annoying and pointless.There are also quite a few books about the thesis of the 'extended mind' (Clark & Chalmers, 1998) ; the idea that cognition quite literally extends into our bodies and environments, in terms of the tools we use and the objects we interact with. It's getting quite hard to find a book on these topics that isn't a) simply rehashing old ground or b) trying to come up with it's own distinct flavour of embodied, extended cognition that fixes some problem of everyone else's. I find this sort of fracturing and endless bickering quite distracting and troublesome, for reasons best expressed by xkcd. I also agree with Tony Chemero, that much of this work is still, sometimes deep down, representational. I think this is because they fail to embrace a sufficiently useful theory of information, such as that proposed by Gibson. So I often find myself reading these books finding nothing new and far too much that's depressing.

Louise Barrett's book, "Beyond the Brain: How Body and Environment Shape Animal and Human Minds" is, refreshingly, not like this at all. Barrett is a psychologist who studies animal cognition and behaviour, and her book does a lot of things very, very well. I'm not planning on a chapter by chapter book club on this, although I may at some point; Eric Charles has posted a few thoughts on the book as well, here, here, here, and here. To cut to the chase: if you are new to the area of embodied cognition, read this book. If you're familiar with the literature but want a clear, well-structured presentation of many of the key ideas, then read this book. If you're bored with the same old examples and want some new, perhaps more convincing examples of embodied cognition in action, read this book. And if you have heard some of the arguments but still think behaviour really comes from the computational activity of our complex brains, then, for the love of science, read this book.

Old stuff done well
One sign of a good book on embodied cognition is that it handles the classic material well. There's a standard set of important papers and empirical results that form the core of the initial case for embodiment and, specifically, embodiment expressed in terms of non-linear dynamical systems. Among these old favourites is the Watts steam governor to replace the digital computer as a metaphor for the way to achieve controlled behaviour (van Gelder, 1995). Like me, Barrett is entirely unconvinced by the computational, representational accounts of the governor and is happy to take it seriously as an example of how specially built coupled devices can solve complex control problems. Her account of it, and how it works as a metaphor to guide how we think about embodiment and dynamics, is crystal clear.

Other topics done well: a look at the various robotic exemplars of the dynamical, embodied approach (such as work by Pfeifer & Scheier (1999) and Pfeifer & Bongard, 2007), plus a nice review of work by Linda Smith and Esther Thelen on the A-not-B error and infant prehension (see Thelen & Smith, 1994).

The coverage is excellent and clear (I've added Chapter 7, on computational information processing vs. dynamical systems accounts of behaviour to my first year Motor Control module recommended reading list as a summary of two lectures).

New things brought to the table
Barrett specialises in animal cognition, and this angle on embodied cognition is, I think, one of the key novel contributions of this book. I admit, I'm a true believer in embodied cognition, so I'm biased, but I found many of the key ideas about embodiment much less controversial when they arose in the context of non-human behaviour. Chapter 4 (entitled 'The Implausible Nature of Portia') describes the simply astonishing behavioural flexibility of the hunting spiders of the genus Portia. These spiders prey on other spiders. They are capable of all finds of feats, of mimicry (strumming other spider's webs to simulate being prey themselves as a lure), and deception (taking advantage of wind blowing on webs and moving under cover of this 'smokescreen'). Being small spiders, however, they of course have tiny brains.

Their most remarkable ability is that they can approach a target web via quite complex detours if necessary; they are able to navigate to a target that is not currently in their view. Their behaviour looks remarkably like 'insight'; they spend time sitting and examining their environment, not moving through it, until suddenly they pick a route as if they had finally arrived at a decision. The spiders, of course, are doing no such thing: there is no mental manipulation of a representation of their route options required. Instead, the spiders are scanning the environment with their very specifically built eyes, and scanning in a very particular way. These spiders have 16 eyes, some of which work like our fovea (high resolution over a small area) and some like the periphery (low resolution but excellent motion sensitivity). Portia scans the environment using these independent, mobile eyes, and experiments show it simply moves to solve a series of local problems, making the route up as it goes. It doesn't need to 'know' anything about gaps or obstacles, it simply moves according to a simple strategy of 'find the horizontal path'. Portia doesn't need a big brain to 'think about things in their absence', because she instead has a sensibly built body - truly embodied cognition.

Other examples include the caching behaviour of scrub jays. and how crickets choose mates according to their song; the 'decision making' is implemented via a simple neural circuit between two ears spaced a particular distance. This work is especially interesting as it includes discussion of Barbara Webb's work on this system; Webb has built simple robot crickets in which she can manipulate the neural circuit and the nature of the ears to confirm that the crickets aren't making decisions, they are responding to signals in a manner dictated by their design; representations not required!

Has she read her Gibson?
One of my requirements for a good book on embodied cognition is an honest discussion of Gibson. The reason is simple: if you want cognition to be a process that spans the brain, body and environment of an organism, you need a theory of how these elements can be coupled together to form systems. In short, you need a theory of perception, and Gibson is really the only candidate up to the task. There are many people trying to do embodied cognition representationally, and this is just an error,as far as I'm concerned.

The good news here is that Barrett knows her Gibson, and explains it well. Chapter 6 is an excellent review of Gibson's theories, and the context in which they arose. It's been a long time since I read something not written by me or Geoff Bingham about Gibson and enjoyed every word; the experience is deeply satisfying. She also ties Gibson's ideas to the work of biologist Jakob von Uexküll, and his notion of the umwelt (roughly, the world as it is experienced by a particular organism). Two organisms can occupy roughly the same space but experience very different umwelts (e.g. the ladybug on my finger and I are having very different experiences).Organisms are sensitive to some but not other aspects of the environment, and taking the organism's perspective on the world seriously is a key part of good biology.Barrett argues convincingly that it should really form part of good psychology too.

Summary
'Beyond the Brain' is an excellent introduction to the logic underlying embodied cognition (why it's a good idea) and to the various candidate mechanisms that support it (how it might work, via Gibson and the architecture of the body). Barrett's background in animal cognition means she has bypassed many of the annoying arguments typically raised by people who only talk about human cognition and who have fallen into the trap of thinking our cognition must somehow be different in kind to these simpler solutions. This has given her a very uncluttered view of the intellectual landscape, and she brings that view to this book. That's not to say she isn't aware of the various annoying arguments; quite the contrary. She spends some time on Adams and Aizawa's 'bounds of cognition' arguments and cuts through a lot of the distracting gloss on their argument that I must admit I had gotten overly concerned with. She also tackles the critique that embodied, perception-action style accounts can only handle movement and things in the now, and can't address 'representationally hungry', 'thinking about things in their absence' problems. The discussion of Portia addresses this; she also describes a simple robot mouse that can, without any memory built into it at all, achieve delayed reinforcement learning. This continues a recent trend in these kinds of books (Chemero's being the obvious other example) of embodied cognition people looking to address not only the new questions our approach asks, but also some of the old questions people think we can't handle. I admire this a lot, and it's a trend I hope continues.

Read this book, and give it to your friends to read.

Prospective Control I: The Outfielder Problem

2011-10-09T14:25:00.001+01:00

A couple of posts ago I raised the distinction between prediction and prospective control. I was trying to make the point that, if you are coupled to the right information, you don't need to be mentally simulating what's happening so you can run this simulation ahead and predict what's coming up. Prediction of this sort is invoked by representational cognitive scientists to cope with things like delays in the nervous system (e.g. Changizi's 'perceiving-the-present' framework). It's a risky business (if you make a mistake, you are suddenly controlling your behaviour according to an incorrect guess), and the better solution is prospective control. This is when you couple your behaviour to information in the world that doesn't tell you about the future, but that, if you use it for a while, will get you where you want to go.

People requested some more on this topic, and so here we go. People also requested something other than the outfielder problem, but I am going to start with this problem because it is still the best example, and I'll get into some more as we go to demonstrate it's not just baseball where this works.

The task
The first step in answering such a perception-action question is a task analysis. What, precisely, is happening in the task and what are the optical consequences of what's happening? In baseball, outfielders are often required to catch fly balls. These are balls hit high in the air, and the outfielder has to move from where they are to where the ball will land in the time it takes the ball to get there from the bat. The ball is typically moving pretty quickly, and is a long way away from the fielder, so it is very small. This last point means some types of time-to-contact information (like tau) aren't available, because the variation in the apparent size of the ball with changes in distance are below threshold. You can, however, learn to catch fly balls pretty reliably; the question is how.

The critical feature of a fly ball is that it moves through the world along a parabola; it is an example of simple projectile motion.

Figure 1. Simple projectile motion follows a parabola

The distance something travels under projectile motion depends on the angle and velocity at the start, the object mass and size and air density, drag and gravity.

Potential solutions

1. Prediction
The prediction solution suggests you take the initial angle and velocity and use those to predict the parabolic path, assuming constant ball size and mass (fair, given it's a baseball) and constant air density, drag and gravity (less fair; drag varies non-linearly with things like spin and wind). If you have access the parabola, you simply read off where the ball will intercept the ground and go there to wait. However, even though you can estimate the ball's behaviour fairly well with some sensible assumptions about drag, etc (Saxberg, 1987a), people don't seem to use this strategy at all (Saxberg, 1987b).

2. Prospective control: Optical Acceleration Cancellation (OAC)
If people aren't acting on predictions generated by an internal simulation of the physics, they must be coordinating their actions using information available during the flight of the ball. The ball is too far away for information such as 'looming' to be of any use. However, there are certain optical consequences when the fielder moves relative to something travelling along a parabola that enable the fielder to refine that movement so as to intercept that object.

Figure 2. Optical Acceleration Cancellation (OAC). From McBeath et al, 1995

If you are aligned with the path of the ball (i.e. it is heading right for you) then the ball will appear to move upwards, slowing down until it reaches it's peak height and then speeding back up as it falls. In other words, the optical projection of the ball is accelerating. If you then move so as to cancel this acceleration, and make it appear that the optical projection of the ball is moving at a constant velocity, your movement will bring you to the right place at the right time to intercept the ball. If you are moving and the ball still appears to be accelerating, you must speed up in order to intercept it; likewise, if the ball is slowing down, you are running too fast and will overshoot. If you cannot move fast enough to cancel the optical acceleration then you will not catch the ball and you should prepare to catch the ball on the bounce.

This strategy has some appeal; it is an intrinsically perception-action strategy (you are required to move so as to produce a specific spatial-temporal pattern in the optics), there is continuous information about whether you are moving correctly, and you end up doing the right thing without ever knowing where the ball is going to land until you intercept it. However, while there is some support for it (most recently Fink, Foo & Warren, 2009) there are some problems. The first is that the evidence suggests people are not sufficiently sensitive enough to optical accelerations to be able to implement the strategy. Second, the geometry that produces the solution only works properly when your path and that of the ball are aligned; in baseball, this is often not the case. Third, as Figure 2 shows, you actually have to cancel out the acceleration of the ball along the slanted projection on the left; this means you have to cancel the acceleration of tan(α), and not α per se (where α is the angle between the ground and the ball as seend from your vantage point). Like tau, OAC is sound in principle, but the lawful relation between the world and the optics simply doesn't range over a suitably wide scope to allow stable behaviour.

3. Prospective control: Linear Optical Trajectory (LOT)
OAC is on the right track, but isn't good enough to support the observed behaviour (namely outfielders catching all kinds of fly balls). McBeath et al (1995) proposed a different strategy that shares the important features of OAC (continuously available information that allows you to control your behaviour with respect to your goal); this strategy is to move so as to make the ball appear to move, not along a parabola, but along a straight line. This strategy is called Linear Optical Trajectory, or LOT (Figure 3)

Figure 3. Linear Optical Trajectory (LOT). From McBeath et al, 1995.

LOT is a 2-dimensional strategy. The outfielder's job is to move and vary both α (the vertical angle to the ball) and β (the angle along the ground) so as to make the ball appear to follow a straight line path. Varying both angles allows you to intercept fly balls when you aren't aligned to the ball's flight path and have to come in from the side. Instead of optical acceleration, the fielder needs to detect when the flight of the ball is optically curved, and they work to null this curvature. People are much more sensitive to this type of information, and it is an indirect way to detect optical acceleration (hence it works to intercept the ball).

Which strategy do people use?
It's looking good for LOT, but it is always an empirical question what information is being used, especially when there are alternatives (Gibson, 1979). The strategies entail different paths to the same location: prediction and OAC suggest that you will run along a straight path at constant speed; LOT suggests you will accelerate then decelerate along a curved path. Prediction also suggests you will simply run to a given location, whereas the prospective strategies suggest you will move according to the information, not the end location.

McBeath et al (1995) collected some simple data from 2 participants; they tracked the fielder's path and also recorded with a camera the optical trajectory that results from these paths. The paths were curved and entailed accelerations, while the optical trajectories were linear, on 75% of the trials. This suggests outfielders typically rely on LOT. McBeath et al also suggested that the fact skilled baseball players often run into walls and catch on the move, rather than rushing ahead of the ball, also work against the predictive strategy.

It's not yet over for OAC (see Fink et al, 2009); and people continue to run experiments showing that either LOT or OAC dominate in varying conditions. However, all the data support the idea that one of these prospective methods is being used, and prediction of any kind is simply not being made.

Summary
The reason I use the outfielder problem is that it is a complex, real world task that is amenable to the kind of detailed optical analysis required to identify potential sources of visual information for prospective control. That analysis has revealed at least two potential perception-action strategies in which people move so as to make the optic array contain a particular state of affairs, i.e. information. These strategies can be distinguished on the basis of the paths they predict people take (McBeath et al, 1995) and how those paths respond to certain perturbations (Fink et al, 2009). Even dogs can use these strategies (Shaffer et al, 2004), and demonstrating that non-human animals solve a task the same way we do is sometimes a useful way to help rule out the kind of complex internal representations of physics psychology thinks humans, but only humans, have. The outfielder problem in particular, and work on prospective control in general, shows how complex behaviour that looks like it requires a complex mental life can emerge by the operation of a simple rule over time (and Eric's discussing these topics with respect to Louise Barrett's book 'Beyond the Brain' on his blog).

References

Fink, P., Foo, P., & Warren, W. (2009). Catching fly balls in virtual reality: A critical test of the outfielder problem Journal of Vision, 9 (13), 14-14 DOI: 10.1167/9.13.14 Download

McBeath MK, Shaffer DM, & Kaiser MK (1995). How baseball outfielders determine where to run to
catch fly balls. Science (New York, N.Y.), 268 (5210), 569-73 PMID: 7725104 Download

Saxberg, B. V. H. (1987a). Projected free fall trajectories I: Theory and simulation. Biological Cybernetics, 56, 159-175. Download

Saxberg, B. V. H. (1987a). Projected free fall trajectories I: Human experiments. Biological Cybernetics, 56, 177-184. Download

Shaffer, D. M., Krauchunas, S. M., Eddy, M. & McBeath, M. K. (2004). How dogs navigate to catch Frisbees. Psychological Science, 15(7), 437-441. DOI Download

Introduce yourselves, dear readers

2011-09-30T15:09:00.000+01:00

Dear readers:

We keep finding out that all kinds of people are reading our blog, when you email us, or when we say 'hey, you should totally read our blog' and you say 'I already do!'. Lots of you don't comment much, though, so we never find out who you are and we're actually quite interested.

So if you read us, take a moment to leave us a comment below. Introduce yourselves, tell us a little bit about what you do and why you're reading us, and, if you blog and whatnot yourself, feel free to leave links for people to find your stuff too.

Also, please feel free to comment on things if you want to; we both see comments on any posts, even the old ones, and we like the attention! Plus we've had some great conversations with some of you who do comment, and begun all kinds of collaborations and things, and one thing we want from the blog is to open some lines of communication on our flavour of psychology that doesn't get a lot of traction in other, more mainstream blogs.

Andrew & Sabrina

Embodied solutions to neural delays: Information and Network Motifs

2011-09-23T15:48:00.001+01:00

One of the bugbears of direct perception is the fact of neural delays. The transmission of signals through the nervous system takes time, and this means that there is a lag between something happening (at, say, the retina) and that event having consequences in cortex, let alone behaviour. In control theory terms, delays in a system can lead to instability in that system's behaviour as you are forced to make corrections that are then incorrect and must themselves be corrected.

It's typically suggested in psychology that these delays are compensated for via computational predictions; the nervous system 'perceives the present' by taking the lagged input and using it as the basis of a guess about what's going on now (e.g. Changizi et al, 2008, plus see this post from a while back). This is a problematic idea: if the perceptual control of actions is based on a guess compensating for a variable time lag, then the stability issue remains, not to mention the consequences for mis-predicting the future. Regardless, it's not really an option for a theory of direct perception, and I want to discuss a couple of options.

Prediction vs. Prospective Control
Prediction is a risky business. Not only is it computationally intensive and time-consuming in and of itself, errors can readily accumulate throughout the process leading to potentially catastrophic failures of the system. The perception-action system is particularly vulnerable to this, as it typically operates in a 'just enough, and just in time' fashion. Recent empirical work has begun investigating the possibility of prospective control. This is a more robust control strategy that entails coupling yourself to currently available information to control action so as to achieve a future goal. Of course, it requires the availability of useful information.

Montagne et al (1999) tested a way to identify which strategy is being used in an interception task. Intercepting a moving object means getting yourself to the right place at the right time. Montagne et al created a simple interception task in which the object to be intercepted started in one of three locations and smoothly moved to a central location.They also varied the initial location of the person's hand, such that it was either at the final location of the target already, or off to one side. If people were predicting, they should simply move their hand to the final location; if they were prospectively coupled to information about the target, they should exhibit movement reversals. (The geometry of the set up meant that the hypothesised information varied over time for certain combinations of initial hand and target location so that using it would make you overshoot and have to correct). They then went looking for (and found) the movement reversal signature, confirming prospective control (see Figure 1).

Figure 1. Movement reversals in interception (Montagne et al, 1999)

Neural architecture and zero-phase lags
Prospective control is showing up in all kinds of perception-action tasks. However the basic problem of neural delays persists: if you couple yourself to an information variable, it still takes time for that information to be available to control action. Perhaps you're still 'perceiving-the-present' but just basing your behaviour on a predicted value for the information for prospective control. How problematic are these lags?

It's possible that they aren't actually much of a problem, thanks to an intriguing embodied solution to the problem which I came across as I work through Networks of the Brain (Sporns, 2010). Vicente et al (2008) investigated how you can establish synchronous behaviour between widely separated regions of cortex via a simple motif (Milo et al, 2002; see also Sporns, 2010). Motifs are simple patterns of connectivity between nodes of a network; for example, three nodes can be connected in 13 distinct ways, and each of these is a motif. These form a kind of alphabet, from which you can build networks with particular properties. Vicente et al demonstrated that a dual dyad motif promotes self-organisation of the system into synchronous behaviour with no lag, even if there are significant conduction delays present between the nodes (see Figure 2). This solution is embodied because the solution doesn't require computation; rather, the system solves the problem by being built in a particular way.

Figure 2. The α and β neurons begin out of sync but with a few spikes remove the phase lag (fFrom Vicente et al, 2008)

Vicente et al demonstrated that this network architecture promotes this type of self-organisation between individual neurons and populations, over a wide range of latencies. The system doesn't entirely eliminate lag, because it still takes time for the synchronised behaviour to emerge after a perturbation; however, the time taken for the system to respond is less than the time you get if you just add up the individual conduction times between the neurons. Given that this is how the delays typically cited by people like Changizi et al (2008) are computed, this paper is an important contribution towards identifying ways to compute the actual lags in the system.

Reverberation vs. Serial Order
The motif solution is interesting, because it changes the focus a little for the way the problem is posed. Clearly, in a system that actually exhibits conduction delays, you cannot instantly ping one end of a connection and get a zero-phase lag, no delay response at the other end. But the motif solution isn't saying this: the fast self-organisation of synchronous behaviour emerges once the system in question is up and running. It takes a finite amount of time for information to zing around the network from the moment you 'switch it on', but once it's going, the network that has been built now reverberates as a whole in response to events in any node. Typically, lag estimates are computed by just adding up the conduction times for all the neurons involved in a linear, serial event; for responding to stimulation on your toe, for instance, you add up the conduction times for the long axons that synapse in the spinal cord and the axons that connect from the spine to cortex (plus time for the slower chemical transmission process at each synapse). Same for the supposed lag between the eye and visual cortex.

The analogy that came to mind is what happens when you switch a light bulb on in a dark room. Light leaves the bulb and streams out in all directions; it travels until it hits a surface, when it is reflected according to the relevant laws. Eventually, some of this light passes through the point in the room occupied by your eye; light at this point of observation comes from all over the room (and hence you can see all over the room) and continually flows through that point. The room is entirely filled but only after a measurable amount of time; it seems instantaneous because light travels so fast.

Once the room is filled, you can simply move through this optic array and the light you sample smoothly alters in response. Critically, light from objects at different distances behaves appropriately; even though the light had to travel different distances and therefore took different times, there is no lag - objects further away don't take longer to 'update' a given point of observation. Why? Because once the room is filled, the light coming from those two surfaces has already covered the distances to the point of observation and is simply being replaced with a continuous stream of light that's done the same. The lag problem has been solved because it happened during the time when the room was filling with light.

I think that the motif solution is a bit like this. The lags the motif is fixing exist, but only until the system is up and running. Once the system is going, information continually flows through, say, a given point of observation and the structure at that point implements the lag correction already. Perception, action and cognition are continuous and are built to take advantage of what that continuity affords.

Summary
Lags and delays in the system are typically a problem which is solved computationally, via internal models which take an input at time t-delay and use that input to predict what the input will be at time t. But prediction is a risky business because if your internal model is incorrect, you will have your face eaten by that bear (or something similar). Solving these problems via internal computational models is also not a legal move for the ecological approach, so the question is, how do we solve it? What else might it be?

In behavioural studies, we make the distinction between prediction and prospective control. The latter typically predicts a pattern such as movement reversals, because the control strategy is to be coupled to an information variable (typically a higher order variable that specifies the property of the world you're interested in) and your behaviour follows the behaviour of that variable over time; a little like the Watts steam governor. People have gone looking for this signature in a variety of interception tasks (typically thought to require prediction); in general, we only see evidence for prospective control (the outfielder problem is another great example of this). There's much more to do here: the eye movement literature, for example, tends to still talk mostly about predictive mechanisms to explain successful tracking behaviour. I hope to get into this empirically over the next few years.

Neurally, you can also address the delay issue without resorting to computations that implement prediction. Network motifs are architectural features that shorten effective path lengths and create functionally related clusters; in short, they promote 'small world' architecture (Watts & Strogatz, 1998). Those clusters, or hubs (like the γ node above) are essential to maintain network-wide information flow; however, they work by facilitating flow in the network, and not by implementing special computations:

Their influence derives from their capacity to connect across much of the brain and promote functional integration, not from special intrinsic processing power or capacity for "decision making". Hubs enable and facilitate integrative processes, but they do not represent their outcome, which instead is found in the distributed and global dynamics of the brain.
Sporns, 2010, p. 125.

As the Watts governor continues to remind us, there are often dynamical solutions to supposedly algorithmic control problems that work via specifically built devices coupled to relevant information, and these solutions don't implement the algorithmic description. This is a different way to think about, for example. the delays and problems the nervous system actually faces, and the resources it has available to address them. These resources aren't computational: they are architectural and informational. They are, in fact, embodied.

References
Changizi, M. A., Hsieh, A., Nijhawan, R., Kanai, R., & Shimojo, S. (2008). Perceiving-the-present and a systematization of illusions. Cognitive Science, 32(3), 459-503. DOI Download

Montagne, G., Durey, A., Bootsma, R., & Laurent, M. (1999). Movement reversals in ball catching Experimental Brain Research, 129 (1), 87-92 DOI: 10.1007/s002210050939 Download

Sporns, O. (2010) Networks of the Brain. Cambridge, MA: MIT Press. Amazon.co.uk

Vicente, R., Gollo, L., Mirasso, C., Fischer, I., & Pipa, G. (2008). Dynamical relaying can yield zero time lag neuronal synchrony despite long conduction delays Proceedings of the National Academy of Sciences, 105 (44), 17157-17162 DOI: 10.1073/pnas.0809353105 Download

Watts, D.J. & Strogatz, S. H. (1998). Collective dynamics of "small-world" networks. Nature, 393, 440-442. DOI Download

Coordination dynamics and relative speed

2011-09-13T12:39:00.000+01:00

The Bingham model of coordinated rhythmic movement makes three predictions. First, it predicts that movement stability is a function of perceptual ability, and we confirmed this in two ways (by showing how people can move stably at non-0° with transformed visual feedback (Wilson et al, 2005) and by showing that perceptual learning of 90° led to improved movement stability without practice at the movement task; Wilson et al, 2010). This prediction is also supported by recent work by Kovacs and Shea, who are busy demonstrating that transformed, Lissajous feedback breaks the classic pattern of movement stability in coordination tasks. The second prediction is that relative phase is specified by the relative direction of motion; we confirmed this by selectively perturbing various components of motion and showing selective effects on performance (Wilson & Bingham, 2008).

The third prediction was that the detection of relative direction was conditioned on the relative speed; the latter was simply a noise term. de Rugy, Oullier & Temprado (2008) tested this prediction by using an amplitude manipulation to alter the relative speeds. Their data did not support the model predictions, and they concluded that the approach taken by the Bingham model was flawed. We recently replicated their experiment (Snapp-Childs, Wilson & Bingham, in press as of Friday; download) and identified numerous critical flaws in their design and analysis which invalidated their criticism.

A problem for the model: de Rugy et al (2008)
de Rugy et al ran a simple study. They had participants control a single dot on a screen using a joystick (similar to our unimanual experimental setup), and track a computer-controlled target at either 0° or 180°. They manipulated the frequency of the required movement (0.75Hz, 1.25Hz, and 1.75Hz) to alter the overall speeds involved. They then altered the amplitude of movement the person had to track so that it was either the same as or three times as large as the person's amplitude, which remained constant. There were then 4 key conditions: moving at 0° with the same or different amplitude, and moving at 180° with the same of different amplitude. Critically, the relative speed in the 0°/different amplitude condition is the same as the 180°/same amplitude condition, and thus these conditions should show similar phase variability. The 180°/different amplitude should then be the most variable condition.

The critical result, however, was that the amplitude manipulation had no effect on movement variability; only the typical effects of frequency and required relative phase had any effect (Figure 1). They measured movement variability using a variable called SDΨ. Relative phase is a circular variable (the distribution of possible values lies on a circle) and there are special statistical methods for handling these kinds of data. One of the variables you can compute is the uniformity, which is a number that ranges from 0-1 and is 0 if the data are evenly distributed and 1 if all the data are the same; the higher the number, the more 'bunched up' the data are. High uniformity therefore corresponds to stable movement, in this case; de Rugy et al applied a standard transformation to this variable to convert it to a number equivalent to a standard deviation (ranging from 0 to infinity with a linear distribution); SDΨ = (-2 log_eU)^1/2.

Figure 1. Data from de Rugy et al (2008)

The rebuttal: Snapp-Childs et al (in press)

This paper poses a challenge to the model. However, that challenge was not actually as great as the authors made out. There were four critical concerns which we spotted immediately:

de Rugy et al used the model to make predictions about the pattern of behaviour in a unimanual version of the task. The model was designed to capture the dynamics of the bimanual version; it includes a bi-directional coupling term which makes this version of the task more stable than the unimanual version. While we have used the unimanual task as a method which allowed us to investigate qualitative predictions of the model, we have very purposefully not fit any of these data using the model.
Performance at 1.75Hz was spectacularly variable; participants could clearly not do the task. This supported our hunch that the unimanual version of the task is less stable, making those data uninterpretable (see the next point).
SDΨ is a common but problematic measure. It only measures movement variability, with no regard to what the person is attempting to do; if a participant is trying to move at 90° and failing, slipping into 0°, then they will typically produce stable but incorrect behaviour. Researchers therefore usually also report the absolute error (which de Rugy et al did not do); all this does, though, is tell you whether SDΨ for that trial is a valid measure of performance. More recently (Wilson et al, 2010a, b) we have used a better solution - we report 'time on task', the proportion of time spent at the target phase, +/- an error bandwidth. This measure validly captures both accuracy and stability, and we have used it with great success.
The amplitude manipulation is only interesting if participants were, in fact, moving at the correct amplitudes. It's well known that this is difficult; people typically assimilate their amplitudes to the one they are tracking (e.g. Kovacs and Shea, 2010). de Rugy et al did not report or analyse the actually produced amplitudes. Maintaining the correct amplitude makes the unequal amplitude conditions a dual task, and thus harder.

We therefore 1) tested the stability of unimanual performance at 180° to confirm that it is less stable, and then 2) replicated de Rugy et al but then analysed the amplitudes produced and added 'time on task' to the dependent variables.

Unimanual movements are less stable than bimanual
Figure 2 shows 'time on task' when attempting to move at 180° with increasing frequency; performance past 1.25 Hz drops and performance past here remains almost entirely uncoordinated. This supported our first concerns.

Figure 2. Increasing frequency disrupts unimanual performance at much lower frequencies than the bimanual version

Is relative speed a noise term?
We then replicated and correctly analysed their experiment. We tested the three key conditions (0°/equal amplitudes, 0°/unequal amplitudes, 180°/equal amplitudes). de Rugy et al's result was that the two 0° conditions were equally stable. However, the model they claimed to be testing does not include amplitude control (part of this design), and if participants are unable to maintain the unequal amplitudes then that condition will stabilise. We asked the following questions:

Question 1: Did the participants maintain the required amplitudes? Only when the amplitudes were equal; in the 0°/unequal amplitude case, they increased their amplitude to be closer to that of the target, thus reducing the relative speed difference.

Question 2: Did the participants produce the correct relative phase? Only in the 0°/equal amplitude condition. In the other two conditions there were significant phase errors, which makes SDΨ an inappropriate measure of performance.

Question 3: Did movement stability decrease as relative speed increased when measured with time on task? Yes. The most stable condition was 0°/equal amplitude, then 0°/unequal amplitude, then 180°/equal amplitude. This ordering makes sense, given that participants reduced the relative speed in the middle condition by moving at the incorrect amplitude.

To summarise: we replicated their results when using SDΨ but when we actually measured amplitude control we found that the 0°/unequal amplitude and 180°/equal amplitude conditions did not produce the same relative speed, and that performance measured correctly reflected this fact. Relative speed is, indeed, the relevant noise term.

Modelling unimanual performance and amplitude control
The de Rugy experiment tested a model of bimanual coordination under equal amplitudes with a unimanual task that entailed amplitude control to maintain different amplitudes. It's therefore no surprise that they failed to find evidence in favour of the model, and the phase and amplitude errors also meant that their analysis using only SDΨ was not up to the task. One of their conclusions, however, was that the basic Bingham model was unlikely to be modifiable to cope with their data; we therefore addressed this as well in the final section. Recall that the reason this model is important is that it is built out of the empirically identified components of the actual system; additions to the model can only use actually perceivable properties, for instance, and not simply add terms to stand in for those properties.

Unimanual performance
We altered the basic model to be unimanual by simply removing the coupling; one oscillator (the person controlled dot) was driven by the perceived phase of the second, computer controlled dot, which was now not driven by phase (the driver was set to 0). We simulated performance at 180° and for various frequencies; like our participants, the model could not maintain coordination past ~1.5Hz.

Amplitude control
The original model does not control amplitude, but does include a parameter, c, where amplitude can be set. We therefore had to find a function that controlled this parameter, using only perceivable properties that did not violate the autonomy of the overall dynamic. The available variables come from the phase plane; we used the radius of the phase plot which uniquely specifies the amplitude throughout the trajectory. This radius is the 'energy' of the system, and there is evidence that people are sensitive to this variable (Goldfield et al, 1993). We modelled the perception of amplitude errors as perceiving the difference between the required amplitude and the current energy of the system.

Relative phase error correction
The model was designed to simulate the task under a 'non-interference' instruction: it's often the case that participants are told not to resist any transitions from, say, 180° to 0°. In this task, of course, people were told to maintain the target relative phase, and the unimanual task with amplitude variation meant people accumulated plenty of errors that needed to be corrected. We modelled phase corrections by using the model's method for judging the current relative phase (integrating Ρ over a 2s window); when this perceived relative phase was sufficiently different from the target, the model simply reset the initial conditions. This matches one correction strategy, where people move to one end and wait for the dot to catch up. We set the 'sufficiently different' threshold to be small for 0° and large for 180°.

The unimanual model which included both amplitude control and phase correction replicated the qualitative pattern of results. While there is actually a lot of work to be done to fine tune these implementations, this served as a proof-of-concept that a model following the perception-action modelling strategy could handle this task.

Why this matters
No other model is built under the rules which demand the entire model be built from actually perceived information coupling correctly formulated action components. This model therefore matters because it represents the future of perception-action dynamical systems modelling. When tested appropriately, the model passes with flying colours; but testing this correctly is hard and requires careful attention to all aspects of the task. de Rugy et al (2008) used the wrong task, the wrong measure, and failed to check the effects of their manipulations. My next project will build on the developments we developed to reveal these facts, and will extend the model to handle frequency and amplitude control, learning and error correction according to the perception-action rules.

References

de Rugy, A., Oullier, O., & Temprado, J. (2008). Stability of rhythmic visuo-motor tracking does not depend on relative velocity Experimental Brain Research, 184 (2), 269-273 DOI: 10.1007/s00221-007-1180-0 Download

Goldfield, E.C., Kay, B.A., & Warren, W.H. (1993). Infant bouncing: The assembly and tuning of action systems. Child Development, 64, 1128-1142. Download

Snapp-Childs, W., Wilson, A. D., & Bingham, G. P. (2011). The stability of rhythmic movement coordination depends on relative speed: the Bingham model supported Experimental Brain Research DOI: 10.1007/s00221-011-2874-x. Download

Wilson, A. D., & Bingham, G. P. (2008). Identifying the information for the visual perception of relative phase. Perception & Psychophysics, 70(3), 465–476. Download

Wilson, A. D., Collins, D. R., & Bingham, G. P. (2005). Perceptual coupling in rhythmic movement coordination – stable perception leads to stable action. Experimental Brain Research, 164(4), 517–528. Download

Wilson, A. D., Snapp-Childs, W., & Bingham, G. P. (2010). Perceptual learning immediately yields new stable motor coordination. Journal of Experimental Psychology: Human Perception & Performance,36(6), 1508-1514. Download

Wilson, A. D., Snapp-Childs, W., Coates, R., & Bingham, G. P. (2010). Learning a coordinated rhythmic movement with task-appropriate coordination feedback. Experimental Brain Research, 205(4), 513-520. Download

Help me write a good lay summary of my coordination research

2011-09-06T12:04:00.000+01:00

I'm writing a grant proposal, and part of the process involves a 4000 character lay summary of the project; what I'm up to, what the objectives are and who the likely beneficiaries are. The goal is for this to be in language a non-expert can understand (Research Councils use these summaries when they promote funded grants to the wider public). I am, frankly, cursed with too much knowledge, and pitching these lay summaries is always a challenge. What I would like is for people, especially non-experts, to have a look at the text below and give me feedback - are there sections that aren't popping? Sentences that aren't clear? Entire sections that seem a bit mysterious?

Please feel free to leave comments pointing out things that aren't working (and things that are, too, actually: it's good for me to get a feeling for what I'm doing right here). If you felt like spreading this around via Twitter, G+, whatever your fancy, I'd appreciate all and any feedback on this!

*****************************

The purpose of this project is to study how we acquire and perform skilled motor activities. Even those of us who aren’t elite athletes perform surprisingly complex actions such as locomotion (walking, driving a car) all the time, and typically with great skill. All these motor activities depend on perception (e.g. vision) for their success – we look where we’re going, and we coordinate our behaviour with others by monitoring what they are doing. Understanding how we perform such skilled motor activities therefore means we must understand the role of perceptual information in supporting these perception-action activities – what kind of information do we use, and how do we use it? A second important fact about our ability to perform skilled movements is that they are learned: it takes time and practice to coordinate the movement of our own limbs to achieve a goal, let alone coordinate our behaviour with that of others. Again, perception plays a vital role, and to study learning we must again ask questions about the information we perceive and how it is used.

The research in the current proposal sets out a method to go about carefully answering these types of questions. The experiments rely on a perception-action task called rhythmic movement coordination, a simple yet surprisingly rich task that has been studied extensively for over 25 years and continues to serve as an ideal test bed for understanding skilled movement. A simple version of the task is as follows: take your index fingers, and move them up and down at the same time; now move them so they alternate. Both of these rhythms are easy to produce, but interestingly, previous research has established that these are actually the only two rhythms people can produce reliably. Try to produce the intermediate, syncopated rhythm; this might be easy if you go slowly, but as soon as you speed up you will likely lose the rhythm (unless you’ve had training, perhaps with a musical instrument). This simple task involves all the key elements we’re interested in understanding (voluntary movement, which is coordinated across limbs via perception) and we can therefore use it to uncover the general principles underlying the performance of skilled actions. In addition, we can also investigate the process of learning novel rhythms and uncover how we learn to use visual information to generate and control new skilled actions.

The outcomes for this project will be an extensive set of data exploring how people use visual information to control coordinated rhythmic movements, and how they learn to produce new rhythms. These data will let us identify the visual information and how, exactly, it is being used. We will then use these data to create a mathematical model of performance in this task. Models provide us with a way to summarise the results of experiments, and to generate hypotheses to guide future research; we will demonstrate this by using the model we develop to predict how people will control their eyes as they visually control their actions.

Answers to these questions about performance and learning have many applications. One example is improving training for elite athletes; another is developing interventions to improve the recovery of motor function after an injury or a stroke (rehabilitation is a form of learning, after all). While coordinated rhythmic movement isn’t a task you’d use in training or rehabilitation, the lessons learned from studying it can directly inform these activities and help improve them in the future. Our research team is already using this task as a tool to investigate how perception-action skills change across the life-span, and the current proposal will support this effort. Older adults often have problems with perception-action skills and consequences (e.g. falls) pose real dangers. Lessons from how older adults perform coordinated rhythmic movements are helping us understand these changes in greater detail, and will help improve interventions.

There's More Than One Way to Rhythmically Move a Lobster

2011-08-23T10:54:00.000+01:00

I'm slowly working my way through Olaf Sporn's excellent book, Networks of the Mind. The purpose of this book is to introduce neuroscientists to network theory, and vice versa; I'm eavesdropping and tooling up on both. It's slow going only because it's pretty much all new territory to me, but I'm seeing a lot of potential in the overall approach to the brain, and this just confirms for me that Sporns understands what he does pretty deeply.

Anyway, a while back, Bruce Hood tweeted the following:

Fact: in the digestive tract of the lobster, its nervous system can have 100k-200k different neural states that produce the same behavior

This struck me as an astonishing fact; to my mind, it throws the idea that hunting for the neural correlates of behaviour into serious doubt. At the very least, it needs to radically change what you expect to find. I finally found the reference for this fact in Sporns' book (Prinz, Bucher & Marder, 2004; download), and Sporns has some interesting context for this fact (which Hood undersells - see below) and why it's interesting, rather than soul-crushingly depressing. There are some interesting potential consequences relevant to my current ponderings on the brain, although it's definitely still at the 'these sound like the same sort of principle' level of analysis.

Inter-individual variation is a fact of cortical organisation; while a given type of brain (e.g. human) exhibits broad structural similarity the specific details are enormously variable. However, these structurally variable brains support functionally similar behaviour - this is referred to as degeneracy:

Structurally variable but functionally equivalent networks are an example of degeneracy, defined as the capacity of systems to perform similar functions despite differences in the way they are configured and connected (Tononi et al, 1999; Edelman & Gally, 2001)

Sporns, 2010, pg. 69

Degeneracy is a strength, not a weakness of a system. It provides critical redundancy to a system, so that it can continue to exhibit a given behaviour in the face of noise or damage (an ability which Sporns also refers to as functional homeostasis). It's also a ubiquitous feature of biological systems across multiple scales, presumably because these continually face the task of performing the same basic function under highly variable conditions.

Degeneracy in the humble lobster gut

The Prinz et al (2004) paper is an astonishing demonstration of the degree of degeneracy available to even a simple neural system. They begin by noting that neuroscience typically treats variation as noise, and removes it by averaging, etc. They were wondering how justifiable this is. To understand the functional consequences of structural variation, they modelled a simple motor circuit in the lobster (technically, the pyloric rhythm of the crustacean stomatogastric ganglion). Combining all legitimate neuron types and synapse strengths gave them 20,500,000 model circuits. They then simulated the motor output of all of these circuits. 4,047,375 (20%) of these produce a pyloric-like output; this was reduced to 452,516 when the simulation was constrained by recordings from lobsters, and even here all 150 possible combinations of neurons were capable of producing pyloric-like activity. Finally, pyloric-like activity occurred for synapse weights across the entire functional range (with the exception of one particular synapse weight which had to be very weak: this, it turns out, matches the biology of the actual system very nicely). In effect, the behaviour of the network (and not just the individual neurons) exhibited extraordinary functional homeostasis (450,000 ways to produce the same rhythm) and the parameters were stable over several orders of magnitude (so long as the appropriate compensation has occurred elsewhere in the network).

The lesson: the nervous system is not trying to maintain a specific set of wiring connections and weights; it is trying to maintain the capacity to perform a specific function. Prinz et al's analysis supports the idea that this function can be defined at the network level, not just at the level of individual neurons (and I'm going to suggest this principle extends to perception-actions systems as well). Sporns then notes that this is clearly a feature, not a bug:

Such homeostatic mechanisms are essential for the long-term stability of the brain given the continual remodeling and structural turnover of its cellular and molecular components

Sporns, 2010, pg.68

This reminds me of something I saw in Sporns cognitive neuroscience class at IU; video of neurons extruding and retracting axons, forming and reforming connections,something they apparently simply never stop doing. I've never forgotten just how busy the neuron was, on a millisecond time scale; structural stability is clearly not what the brain is interested in. There is, therefore, tremendous and ongoing flux that is part of the normal operation of the brain (let alone what happens following injury) and this degeneracy provides this dynamic system with functional stability in the face of these perturbations.

Some implications for the brain

This implies a couple of things. First, the brain is not static in any way, not even the ways we first thought might be necessary to preserve stable function. The brain is in the state it's in through use, and that use can produce all kinds of variability that don't affect function.

Second, this variability will be both within and between people; Sporns again:

Price and Friston have noted that human brain networks display degeneracy since different sets of brain regions can support a given cognitive function (Price and Friston, 2002). Cortical activation maps obtained from functional neuroimaging studies of individuals often show only partial overlap for a given cognitive task, suggesting that different individuals utilize different (degenerate) networks.

Sporns, 2010, pg. 69

While this doesn't make neuroimaging useless, it is an important caveat that is rarely remembered. This kind of variability is also the reason that there are limited lessons to be learned from clinical cases; while these are interesting and do provide hints as to what regions of the brain are involved in, but

The loss of a subset of all regions that are reliably activated in a given task may not disrupt task performance, indicating that individual regions may not be necessary or that recovery processes following brain injury can configure structurally different but functionally equivalent networks.

Sporns, 2010, pg. 69

It's not clear to me that much of cognitive neuroscience is worrying about this, although clearly people like Sporns are.

Degeneracy in perception-action systems

This organising principle (preserve functional outcome, implement it with whatever you've got) actually extends to the level of the perception-action system. Take a task like reach-to-grasp (prehension); you can implement the functional outcome of transporting your hand from where it is to the object via an indefinitely large set of movements. We typically produce very efficient, very stereotyped prehension movements to a given target, but that's only true when the system's resources remain the same. If you change any part of this task (the distance to the object, the size of the object, the goal of the prehension movement) the prehension movement will rescale itself to match and thus preserve the outcome. My colleague Mark Mon-Williams talks a lot about end-point control - prehension is organised with respect to the control of the hand, not the intermediate joint configurations, and those simply do what's required.

A recent article demonstrated this type of principle. Castro et al (2011) investigated what the system is learning when it corrects errors. There are two options: adapt your next movement as a function of the error with respect to what you planned to do, or as a function of the error with respect to what you actually did. Research in this field typically assumes the former (e.g. Wolpert et al, 1998); Castro et al demonstrated that the system actually does the latter, and, interestingly, the robotics literature settled very quickly on these types of algorithms:

The choice of these learning rules is grounded in the idea that adaptive changes should be provably stable in the sense that, under a set of reasonable assumptions, updated internal models should never result in worse performance

Castro et al, 2011

In essence, this work shows that we learn about the functional outcomes of actions, and not about the implementation of the actions. Learning organises the perception-action system with respect to what happened.

There are other examples of this. For instance, in my own work (Wilson et al 2005a, b) I've had people produce coordinated rhythmic movements when I've transformed the display to show a different rhythm. Specifically, we had people move at 90° and 180° to produce 0° on the screen; this stabilises the typically unstable movements almost entirely. In other words, people were more concerned with producing the outcome (0°) and less concerned with how they were doing it - functional outcome trumps implementation. Prism adaptation studies demonstrate the same principle.

This helps explain why, when studying perception-action, we typically aren't that interested in the movements themselves. Instead, we use the movements as action measures of perception (skilled action is typically fairly stable and only changes when the system perceives a difference in the task requirements). Simply reporting the kinematics typically tells you very little about what the system is up to - after all, it either succeeded or it failed, and reporting the specific implementation is only interesting if you have some reason to expect a specific solution.

Summary

Degeneracy seems to be an organising principle that extends across multiple systems and levels in biology. The specific structural implementation is rarely that informative: it may simply be the solution evolution happened to find first, by chance. The real question of interest is, what is the function performed by that system, and how does it relate to the task? This makes a task-specific approach all the more important, and re-emphasises my point that in order to understand the brain, we have to first understand what it is it needs to be up to - the job description is critical, and changing that (as the ecological, embodied approach does) has implications for neuroscience that have yet to be fully acknowledged.

References

Gonzalez Castro LN, Monsen CB, & Smith MA (2011). The binding of learning to action in motor adaptation. PLoS computational biology, 7 (6) PMID: 21731476 Download

Prinz, A., Bucher, D., & Marder, E. (2004). Similar network activity from disparate circuit parameters Nature Neuroscience, 7 (12), 1345-1352 DOI: 10.1038/nn1352 Download

Wilson, A. D., Collins, D., & Bingham, G.P. (2005a). Perceptual coupling in rhythmic movement coordination: stable perception leads to stable action. Experimental Brain Research, 164(4), 517-528 DOI Download

Wilson, A.D., Collins, D., & Bingham, G. P. (2005b). Human movement coordination implicates relative direction as the information for relative phase. Experimental Brain Research, 165, 351-361. Download

Wolpert, D., Miall, R., & Kawato, M. (1998). Internal models in the cerebellum Trends in Cognitive Sciences, 2 (9), 338-347 DOI: 10.1016/S1364-6613(98)01221-2 Download

Mirrors are Literally Windows to Another World

2011-08-16T14:21:00.003+01:00

The other day, psychologist Tom Hartley tweeted "Your reflection is always half the size of the real thing - no matter how far from mirror. Hard to believe but true." and linked to this post in which someone demonstrates this effect. I had never quite thought about it, but realised it was of course always true: the mirror is at half the distance specified in the reflection. Then I read this post linked from the original, which reviewed an article by Lawson et al (2007) describing how people misjudge the size of objects on mirrors - specifically, they think the projection is larger than it actually is. This got me thinking about some work by Gibson on slant perception (Gibson, 1950) and then I realised that this really is an interesting topic. So I'm taking a break from the brain this week to blog about some optics instead.

People's understanding of reflection

In the world, we typically exhibit size constancy: we perceive a given object to be the same size regardless of it's distance from us, even though it's angular size and projection on the retina shrink with distance. Critically, you need information about the distance to the object for this to work reliably, and in traditional cognitive approaches you must combine the two cues to size and distance to achieve your percept of the object. Ecologically, of course, there is typically extensive motion based information for distance in the behaviour of the object in question (e.g. this demo based on Jokisch & Troje, 2003 - an excellent paper, by the way) so there's no cue combination required.

People also show size constancy for objects reflected in mirrors when they are treated as objects at the specified distance - this is useful, because it allows for cheap virtual reality (e.g. Coats et al, 2008). Lawson & Bertamini (2006), however, showed that people lose this size constancy when asked about the size of the projections in mirrors; the 2D structure at the surface of the mirror. People robustly over-estimate the size of the projection - in other words, people don't see that the projection is, in fact, only half the size of the real object. Lawson et al (2007) claim that this suggests there is no percept for the projection: it isn't a distal object and thus isn't perceived as such.

The Experiments

Lawson et al (2007) conducted some experiments to compare people's judgements of the size of a physical object (bamboo sticks) and the size of the projection of the object, and compared performance using mirrors and windows (which has a faint image on it's surface in the right lighting). The stick was held over the observer's head in the mirror condition, and held by an experimenter at the matching distance (2m) in the window case. People were asked to judge the physical size of the stick, and the size of the image of the stick in the mirror or window. The results showed that physical size was judged quite accurately, but that the projection size was widely overestimated in both the mirror and window cases. People rated the projection to be about 80-85% of the physical size (remember the correct answer was 50%).

They next varied the distance; They had people stand at 1.5m or 6m from the mirror or window. This had no effect on the over-estimation of the size of the projection or the essentially accurate judgements of physical size. Next, they made the distance of the observer to the mirror/window different to that of the bamboo stick (to make the actual ratio of projection size:physical size closer to .86). People's judgements of physical size remained accurate, and their judgements of projection size did not change, and were thus 'accurate' in the varied distance condition. This last actually means that participants aren't perceiving the projected size at all, because otherwise their judgements would have changed in response to the size of the projection.

Lawson et al (2007) conclude that people cannot perceive the projection on the surface of the mirror or window, and instead only perceive the distal object that is at some distance 'into' the mirror. Roughly, people aren't overestimating the size of the projection: they are only perceiving the size of the physical object and then combining this with some other knowledge about the presence of the mirror, dragging the estimate down. People only perceive objects, and not mere projections.

Gibson and slant perception

The basic result here is that people correctly perceive the 3D structure of the world 'in' the mirror, and don't perceive the 2D structure 'on' the mirror. When asked about the latter, they systematically misjudge the size of that 2D structure, in the direction of the 3D structure. This pattern reminded me of a study by Gibson (1950), who was studying the perception of visual surfaces (this is early work on his proposal that we don't perceive space, we perceive surfaces, and then, in his later work, what those surfaces afford). One of the experiments involved judgements of optical slant, which is an angular measure of a surface's deviation from the perpendicular to the line of sight (0°). A surface which is slanted relative to the line of sight extends in distance, and presents a texture gradient to the observer - the optical size of the texture elements at the close end are larger than those at the far end, but people exhibit size constancy and, instead of perceiving the projected size, instead perceive the correct physical size and that these elements are varying in distance (as in Lawson et al, 2007).

Gibson presented observers with vertical surfaces (i.e., 0° slant) painted with optical texture gradients specifying a non-0° slant. The displays were viewed in a headrest and monocularly, to reduce the motion and binocular information specifying that the surface was actually vertical. Participants then used a palm board to reproduce the visually specified slant. While their judgements covaried with the visually specified slant, they still systematically underestimated the slant. Essentially, they responded to the texture gradient information but were also sensitive to the fact that this gradient was actually on a vertical surface.

I think this kind of analysis helps firm up the otherwise weak conclusions drawn by Lawson et al (2007). They are basically correct in their explanation, but they haven't done an analysis of the optics and I think this would help make the whole thing less vague. It ties into my big gripe about perception work, not doing the information because it's hard.

The optics of reflection

Reflection is a surprisingly complicated effect; it's explained in clear fashion on this Ask A Mathematician/Ask a Physicist post (a great blog, by the way). One of the interesting things is that, for analysis purposes, reflection is best explained by pretending the surface isn't there and describe the reflection event in terms of an identical (but negative) wave travelling in the opposite direction. This figure demonstrates the analysis, and the figure caption is especially useful:

"Mathematically, as long as you restrict your attention to the left side [your side of the mirror], the following cases are exactly the same: 1) a wave reflecting off of a non-permeable surface and 2) no surface, but two waves, one positive, one negative, passing each other such that they exactly cancel."

So when you are viewing yourself 'in' the mirror, it is as if the light has, indeed, travelled twice the distance from you to the surface of the mirror. The optical structure, therefore, reflects (no pun intended) having travelled that distance. The speed at which optical flow elements move varies inversely and linearly with distance - the further the object, the slower it appears to move (motion parallex). When you move in the world, your reflection 'in' the mirror moves as well, and optically, moves in a manner appropriate to the distance 'in' the mirror, rather than the distance 'to' the mirror itself.

Optically, then, the setup is this: you are at, say, 2m from the surface of the mirror. Your reflection is light that has been structured as if it is 4m away from you (2m to the mirror, 2m 'into' the mirror') and it flows in a manner that is entirely correct for this longer distance. Your projection is the projection of your reflection onto the surface of the mirror; the reflection is 2m 'into' the mirror, and the mirror is only 2m from you, so this projection is always half the size of the real thing (you) and of the reflection. The reflection moves and changes in a manner that is perfectly consistent with an object 4m away from you, and you are thus able to show size constancy about the physical size of the reflections (motion and parallex are very informative about distance and thus support the constancy without the need for you to 'know' the size of the object; eg Jokisch & Troje, 2003). So the physics of reflection mean that the world in the mirror behaves, optically, exactly as it should for the distance it appears to be. Lawson et al (2007) then demonstrate that it is this flow structure is what we perceive, and not the projection structure.

That said, you can ask people to make judgements about the projection structure. The current results suggest, however, that people are staggeringly insensitive to this projection structure, and while they can generate a response, that response does not change with changes in perspective structure. So people aren't overestimating projection size, regardless of the article title; they are actually generating responses based on the reflection structure and producing underestimates.Why underestimates? Because while the reflection structure specifies an object at 4m, there is also information that the mirror is actually at 2m, and while the 3D reflection structure dominates, the 2D surface information also seems to matter.

What is this surface information that tells people about the distance to the mirror? The mirror is not simply hanging in empty space, nor is it taking up your entire field of view. The mirror has an edge, and edges are highly salient for vision because they are where the flow of optical texture exhibits a discontinuity. As I mentioned, motion parallex is the fact that optic flow changes speed smoothly with distance. Edges occur at the end of surfaces, and if this surface is in front of another surface the texture on the nearer surface flows faster than the texture on the far surface; at the edge, this shows up as an abrupt discontinuity in the speed of the optic flow.

In the mirror, the world appears to continue on into the mirror and the optical structure is appropriate for that; within the bounds of the mirror, optical distance increases. In contrast, the wall the mirror is mounted on remains at a constant distance (2m) from you. And, importantly, the edge of the mirror is 2m away and the optic flow discontinuity specifies this fact. You now have conflicting information, that the object being judged is simultaneously 4m and 2m away. It's apparent size is correct for 4m, but if that same apparent size was at 2m, it would appear to be a smaller object (size constancy).

Cue-combination - what might the ecological equivalent look like?

The real question is why does the final judgement look like some kind of weighted average of these two distance measurements (weighted in favour of the stronger, more robust reflection structure)? Traditional cognitive views would see this as a post-perceptual process, similar to how you get size constancy, where you perceive these conflicting cues and resolve the conflict by some process of cue-combination (using the latest in sexy statistics to make the result useful). Ecologically, this cue combination should really be happening in the world; the structure of the light should instantiate the combination. But given the geometry I've laid out, its not clear to me how or why this could work. Perhaps it's the answer you get when you ask a system calibrated using the correct optic flow and motion parallax a question about something which violates the form of that calibration: not quite random nonsense. I'd be interested in experiments which recalibrated the rate at which optic flow speed varies with distance and looked for consequences. But to be honest, I don't quite have an explanation for cue-combination, other than it's a hint we've asked the wrong question.

Summary

This foray into mirrors makes clear (I hope) the idea that we do not perceive light, or structure in light. We don't perceive the light as on the surface of a mirror (where it really is), but instead we perceive what the structure in light specifies. The reflected world in the mirror doesn't really exist; it is a purely optical structure. But we happily perceive it as a normal world of the specified size, because it behaves perfectly appropriately. The only transformation it does impose (the left-right reversal) has no effect on the structure and behaviour of the light, and thus no effect on our perception of the world specified by that light. There is, to the visual system, no difference between looking in a mirror and looking out a window. The only weird part is what happens when you ask about the projection on the mirror itself, and the data clearly support the conclusion that our perception of the reflected world is not based in any way on this projected image - variations in projected size simply aren't perceived and asking about the projection simply produces noisy results derived from the perception of the extended mirror world and the actual distance of the mirror. Cue combination? Maybe; the judgements aren't random. But the answer, as ever, lies in careful analysis of the optical structure in the task, and this must form a critical part of future work on this topic.

References

Coats, R., Bingham, G.P. & Mon-Williams, M. (2008). Calibrating grasp size and reach distance: Interactions reveal integral organization in reaching-to-grasp movements. Experimental Brain Research, 189, 211-220. Download

Gibson, J. (1950). The Perception of Visual Surfaces The American Journal of Psychology, 63 (3) DOI: 10.2307/1418003

Jokisch, D. and Troje, N. F. (2003) Biological motion as a cue for the perception of size. Journal of Vision, 3, 252-264. Download

Lawson, R., Bertamini, M., & Liu, D. (2007). Overestimation of the projected size of objects on the surface of mirrors and windows. Journal of Experimental Psychology: Human Perception and Performance, 33 (5), 1027-1044 DOI: 10.1037/0096-1523.33.5.1027

What's the difference between perception and conception?

2011-08-12T13:46:00.000+01:00

As Andrew has been tackling a new job description for the brain (part 1 and part 2), several comments have been made that suggest that his approach (and the ecological stance in general) might be fine for perception/action, but not for other types of tasks/behaviours. Later on in this post I also think about how we might be able to distinguish between association and direct access to meaning, which is another idea that has been coming up repeatedly in the comments (see here).

In this post I want to think about what makes perception different from conception (Andrew reviews William James' views on this distinction here). I will argue that both occur as networks of evolving neural activity (with perception, this network extends to the environment and the body), but these networks have different properties because they are driven by signals of differential stability. I do not think it is accurate to think of perception as something that happens in V1, for example, and conception as something that happens in SFG. If parts of the brain reverberate in a system along with parts of the brain that are directly interfacing with an energy array in the environment, it seems correct to refer to that whole system as perception. In contrast, if a network of activity appears to be relatively encapsulated with respect to external energy arrays, it seems correct to refer to that system as an act of conception. For now I will leave aside the problem of how conceptual networks develop, but I hope to return to this later.

By this characterisation, a conceptual network is identical to a perceptual network except that it is not being driven and constrained by the detection of perceptual information. This avoids a dualism between perception and conception while providing a basis for predicting different properties of the two systems.

In principle, it should be possible to distinguish perception and conception by analysing the properties of networks of neural activity. In practice, of course, we lack the tools to do this (although mathematical tools do exist that can distinguish between properties of different types of networks). Still, it is worth considering what would be the expected characteristics of perceptual vs. conceptual networks if we could peer inside the brain to observe them.

Perception: Perception involves directly interfacing with energy arrays produced by the environment. This perceptual information is structured and continuous. As the information hits some type of receptor (e.g., in the retina) the energy interacts with specialised cells to generate structured neural signals. I have no idea what the fidelity is between perceptual information and the neural signals it produces, but I think it's safe to say that there is a tight coupling between perceptual information generated by the environment and the resulting neural signal (if signal conjures too linear a picture, think of a reverberating neural system). The tight coupling between environment and neural activity must, to some extent, be communicated to the body to permit us produce flexible, appropriate actions in a dynamic environment.

Perceptual networks should be driven and constrained by the conversion of perceptual information in an energy array into a structured neural signal. In an action task, there will be additional information arising from the perception of our bodies as they perform. The structure provided by this interface with the external environment (and due to the fact that our bodies also possess relatively stable properties) should make perceptual networks relatively coherent and stable and they should reflect the dynamics embodied by the perceptual information driving them. The particular characteristics of a perceptual network will also be influenced by the history of activating the network (i.e., long-term changes in the brain that support a particular type of network behaviour), whether the network has been activated recently (i.e., medium-term changes in proximate network activity such as calibration), and other networks of activity (perceptual or conceptual) that might impinge.

The neural activity resulting from the interface between brain and structured energy arrays in the environment will continuously evolve in response to changes in perceptual information. In this sense there are not discrete networks that activate identically each time a given type of information is present. Instead, the present network will be pulled into a different state to reflect changing environmental conditions.

To the extent that the brain is not entirely single minded, the perceptual neural system can be impinged on by other networks of activity. There will be threshold effects such that the system is resistant to perturbations, except those of sufficient intensity. For example, if I am very pre-occupied with my growing to-do list at work, I might not pay as much attention to avoiding other pedestrians when walking down the street. But, if I am only a little concerned with my to-do list, then my walking performance will be unaffected. There will also be degree effects such that the system can either be influenced (but retain its essential characteristics) or interrupted (you are no longer a device that can perform that particular perception/action task). Thus, there are two predicted hallmarks of a perceptual system of neural activity: 1) some direct interface with an energy array and 2) stability against influence from other networks.

It is worth noting that this characterisation applies to a trained system - one that has already learned to effectively detect the relevant perceptual information (reflected in the long-term changes mentioned above). During learning, a source of instability in a perceptual network will be the organism's ability to reliably perceive the information variable.

Conception: Dreaming is one of the purest examples of conception. The perceptual world intrudes on dreams from time to time but in dreams we are more cut off from perceptual information than at any other period of state of consciousness. Compared to waking life, dreams are highly unstable. Locations appear to change in an instant, events do not necessarily follow any sensible causal sequence, impossible things occur.

Leaving aside these qualitative observations, what would be the expected properties of a network that was not directly interfacing with a stable external environment? At a guess, it seems like the stability of such a network would depend on whether or not a person has a history of activating this network (again reflected in long-term neural changes as a consequence of learning), whether the network has been activated recently (medium-term changes reflecting residual activation of the same or similar networks), and whether there are other competing networks of activity that might impinge. Without structured neural activity driven by the perception of perceptual information, conceptual networks loose an important source of stability. Even in the highly practiced domain of language, consider the differential stability of talking to oneself (without sub-articulating) and speaking aloud. Even so, language confers some important stability to conceptual networks, in that we can use words to maintain an idea in the absence of external perceptual information about that idea. Generally speaking though, conceptual networks will be relatively susceptible to perturbation. Thus, the predicted hallmarks of a conceptual system of neural activity are: 1) no direct interface with an external energy array and 2) instability to interference from other networks.

If it were possible to peer into an active brain and observe its activity in great detail, it would be possible to distinguish between perception and conception on the basis of the above descriptions. Although this is not possible, I think it is worth considering this distinction in light of another issue - how can we tell whether we have direct access to meaning (DAtM) via perception or whether the detection of perceptual information is associated with a meaningful representation? This issue has cropped up several times in comments to previous posts. Andrew and I have both expressed some concern about what type of evidence could distinguish one claim from another. Are they simply different ways of saying the same thing?

I will attempt to outline how these accounts might differ in the language of perception and conception described above. At this point I am not completely confident that I am not just re-describing the problem. But, I think there might be an opportunity here to use network metrics to disambiguate DAtM from association.

Association implies two or more things paired together (a representation of a tone becomes linked to a representation of food), whereas DAtM implies one thing with a higher order interpretation (tone means food). On the basis of these crude descriptions it seems like one could sensibly predict association and DAtM to be distinguished by their instantiation in networks of neural activity. Take a case where a rat has been extensively conditioned to expect a food to follow presentation of a tone:

For association: During training, the rat would experience two consecutive, independent networks of perceptual neural activity - one for the tone and one for the food (I'm obviously oversimplifying by referring to a single network of activity about food, but I want to avoid details like that for now). The activation of the tone network reliably predicts the activation of the food network. Overtime, basic associative neural learning mechanisms begin to reflect the statistical reliability of the tone / food pairing. In a trained rat one would observe a coherent system of perceptual neural activation (perception of a tone) causing the activation of a separate system of conceptual neural activation (thinking about food). These two systems should be relatively independent, except to the extent that the perceptual system causes the conceptual one to fire.

For DAtM: During training the rat would experience the continuous evolution of a perceptual network that is first driven by auditory tone information and is subsequently driven by visual and taste information about food. In the interim between tone and food the network would begin to loose coherence, but stability would return with the presentation of visual and taste information from the food. As I argued earlier, I do not think it makes sense to talk about discrete perceptual networks that fire the same way regardless of what else is happening. I think it is more appropriate to focus on transformation from one type of network into another based on perceptual demands. By this account, the rat is experiencing a single compound event (food -> tone) with a predictable dynamic. A key idea of event perception is that experiencing part of the event can give you direct access to the rest of that event. So, for example, hearing a bar of your favourite song gives you access to the rest of that song, not through association but by the activation of a single event structure with a coherent spatio-temporal dynamic. If this logic is extended to the current example, for a trained rat, experiencing the tone could give him direct access to the entire tone -> food compound event. In contrast to the association case, one would not observe two independent networks, but the evolution of a single network.

Still, one might argue that these two cases are very similar and might be indistinguishable. Both start with a network of activation reverberating with the perceptual information of a tone. But, the DAtM case involves the activation of a single (albeit complex) event structure. Therefore, the observed activity should exhibit the characteristics of a coherent perceptual network. On the other hand, the association case involves the activation of a perceptual network and an independent conceptual network. Thus, the observed activity will differ from the DAtM case because a conceptual network about food will be relatively unstable, susceptible to perturbation, and disconnected from an interface with the environment.

This is a very early first swing at the problem and I am admittedly neither a neuroscientist nor an expert in complex networks. I have tried to make minimal assumptions about both subjects in the hope that what I've suggested is at least somewhat plausible. To a large extent, I agree with Andrew saying that he's tired of having the argument and would like to move onto thinking about new positive ways to approach the problem. But, I'm also a little bothered by the idea that we can't figure out what type of evidence distinguishes an ecological account from a representational one, or a DAtM account from an associative one. So, in this post I've tried to start at the bottom, and hopefully later I can work my way up to making sensible predictions about what behavioural differences we'd expect.

Mirror Neurons, or, What's the Matter with Neuroscience?

2011-08-09T17:05:00.003+01:00

One of the problems I face as I try to figure out what the brain is up to, if not representing, is that I can't rely on the neuroscience literature to back me up. The problem is that, while there has been a lot of data collected over the years, very little of it has been collected within an ecological framework. Neuroscientists are looking for how the brain represents information, not how it perceives it; they're looking to see where perception and action are integrated in the brain, not how the brain-body-environment system produces stable, functional behaviour. This matters because there's no such thing as theory-free observations - all data comes from this experiment rather than that experiment, and even simply reporting a result is laden with theoretical assumptions, even when these aren't explicitly identified. So until I can find a neuroscientist interested in collecting a little data (and I would love to hear from any such person!) I'm limited to laying out the consequences of taking the ecological route and critiquing what's out there already.

The worst offenders, in terms of theory-laden data disguised as 'merely the observed facts', are mirror neurons. They are, I think, the text book example of what's the matter with neuroscience, and I thought it was about time to talk about them a little. For those interested, the most recent exposition of what mirror neurons are and do can be found in a recent (and currently open access) issue of Perspectives on Psychological Science: there's a brief introduction (Glenberg, 2011a), a detailed Q&A paper (Gallese et al, 2011) by five main figures in the field, and a summary (Glenberg, 2011b).(Update: recent meta analysis finds mirror type activity all over the brain, including the cerebellum!)

Mirror Neurons

Mirror neurons were discovered by accident in a monkey lab in Italy (Di Pellegrino et al, 1992). While measuring single neurons from premotor cortex in macaque monkeys, someone noticed that these neurons responded both when the monkey was moving to pick up a peanut and also when the monkey was watching someone do the same action. Why would a single neuron in motor cortex respond to a merely perceptual event?

The suggested answer is that these neurons must be how the brain manages to reunite two separate processing streams (perceptual and motor). Perception is input; signals flow in from various sensors (e.g. the retina) to anatomically distinct regions of cortex, and these are processed into increasingly higher-order information as the signal is passed up at least two streams (dorsal and ventral, for visual information). Motor control is output; top-down motor commands are issued from motor cortex, generated by schemas and programmes designed to produce classes of movements. Perception and action are clearly distinct from one another, both in terms of function (input vs. output) and in terms of anatomy (cortical and bodily). We can do some interesting things, however, that seem to transcend this obvious boundary: we can copy someone else's movement, for example (imitation) and when we observe someone moving we can interpret and understand what that movement is for (action understanding). Mirror neurons seem to be the solution to the problem of how this could possibly be: if there are neurons tuned to both perception and action then there is a location in the brain where these streams cross, so to speak.

Mirror neurons solve a problem that doesn't exist

The main problem with mirror neurons is that they are solving a problem which doesn't exist; perception and action are not separate domains which must somehow come into alignment. There is, instead, the single domain of 'perception-action'; this system is how we interact functionally and successfully with the environment. The reason neuroscience looks to mirror neurons as so wonderous and amazing is that it looks to anatomy and says 'these two things are separate, we must find out how they talk to each other'. One of Gibson's many insights is that anatomy is not the final arbiter here, function is. Perception-action is the function of a broad, integrated system, not the labels of distinct entities. So 'motor' cortex responding to 'perceptual' input is only astonishing if you believe these are two separate functions of the system.

So what are mirror neurons doing, then?

I don't dispute that Rizzolati found what he described: single neurons responding to both perceived action and the action itself. I'm disputing that this means these cells are resolving a Cartesian dualism. I'm not actually surprised there are cells that respond this way. The control of action, which 'motor' cortex is presumably involved in, also includes the ongoing perception of what the limb is up to. There is then significant overlap in the perceptual content of 'my limb moving towards a peanut' and 'your limb moving towards the peanut'; they are, in fact, two instances of the same event.

Event perception

We can categorise what goes on in the world into events. Two events are of the same type if they contain the same information, and they will only overlap in terms of information if they overlap in terms of the dynamic description of what's going on in the world. For example, a pendulum event only occurs when something is moving according to a particular equation of motion. Two instances of a pendulum event can vary in their parameters (e.g length, and thus period) but the structure of the motion will be the same. Information about these events is specific to the stable dynamic structure, not the varying parameters, and so these two events will be identifiable as being of the same type. Different events have different spatio-temporal structure, with different informational consequences, and can therefore also be identified as different on this basis.

We are actually extra-ordinarily good at this sort of thing (Muchisky and Bingham, 2002), and from very early on, although it of course requires learning (Wickelgren & Bingham, 2001). For example, you can make a point light display of a ball being hit, accelerating then slowing down to a stop. Adults can tell the difference between the display run forwards or backwards (the velocity profiles are not symmetrical) and the backwards one simply looks incorrect (because it is - it would never occur naturally). Babies can tell the difference between the two (demonstrated via habituation) but have no preference for either one - they have yet to learn what the difference means (Wickelgren & Bingham, 2001). Biological motion perception is another fascinating example of event perception.

In the case of you or I reaching for that peanut, that same idea applies: these two motions are instances of the same underlying dynamical process, and thus overlap in their perceptual consequences. The overlap is not 100% - you can tell the difference between the two events. But they do overlap, and this overlap is meaningful - specific to the dynamics of the event. This overlap also supports an interesting feature: perceptual constancy, or view invariance. People can readily identify an event from any view which preserves access to the relevant motion profile (although some views are obviously clearer than others; Wickelgren & Bingham, 2004; Zaal et al, 2000).

Event perception is grounded in information, not mirror neurons

One of the things I struggle with a bit at the moment is that much of what I'm trying to do simply sounds like I'm redescribing an already explained phenomenon (see the final note on this post). Not here. Everything I've said explains why we might treat an instance of me moving as similar to an instance of you moving, and this explanation is based in the information created by the underlying dynamics of the event type. You don't need specialised neurons whose job it is to knit these things back together: to identify what you did as similar to what I did I simply need to perceive the informational overlap. One of the much touted jobs for mirror neurons (action understanding) becomes perception-based, and similarly, the other (imitation) becomes about moving so as to produce the underlying dynamical structure, as specified by the view-invariant information specific to that event.

But these single neurons are up to something - so what? While there have been moves in the literature to talk about 'mirror systems', Rizzolati's original discovery was the dual sensitivity of a single neuron. However, single neurons simply aren't responsible for anything. Every neuron in our brain is actually an active participant in multiple networks; these networks ebb and flow over time in response to use, and what a single neuron's behaviour means depends entirely on it's current network context. Rizzolati's neuron might simply have been involved in event perception, and the astonishing thing isn't that it was found in supposedly 'motor' cortex. No, the astonishing thing is that we persist in calling that bit of the brain motor cortex given this data. The error, of course, comes from the unwarranted assumption that perception and action are two distinct domains.

Summary

The problem with mirror neurons is that they are nothing of the sort; they do not implement a connection between perception and action in motor cortex which grounds action understanding, imitation, empathy or lack of autism. The fact that they tend to be in 'motor' cortex only means that we've labelled that bit of cortex badly, and given the fact that all actions entail perception-action, I'm not actually surprised it is interested in event structure.

To my knowledge, no-one has done "mirror neuron" research using event forms as controlled stimuli, although that would make an excellent beginning to an ecologically based neuroscience. Only when we have the right job description for the nervous system, however, will we understand what it's up to when we give it something to do.

References

Di Pellegrino, G., Fadiga, L., Fogassi, L., Gallese, V., & Rizzolatti, G. (1992). Understanding motor events: A neurophysiological study. Experimental Brain Research, 91, 176–180. Download

Gallese, V., Gernsbacher, M.A., Heyes, C., Hickock, G., & Iacoboni, M. (2011). Mirror neuron forum. Perspectives on Psychological Science, 6, 369–407. Download

Glenberg, A. M. (2011a). Introduction to the mirror neuron forum. Perspectives on Psychological Science, 6, 363-368. Download

Glenberg, A. M. (2011b). Positions in the mirror are closer than they appear. Perspectives on Psychological Science, 6, 408-410. Download

Muchisky, M.M. & Bingham, G.P. (2002). Trajectory forms as a source of information about events. Perception & Psychophysics, 64(1), 15-31. Download

Wickelgren, E. & Bingham, G.P. (2001). Infant sensitivity to trajectory forms. Journal of Experimental Psychology: Human Perception and Performance , 27(4), 942-952. Download

Zaal, F., Bingham, G., & Schmidt, R. (2000). Visual perception of mean relative phase and phase variability. Journal of Experimental Psychology: Human Perception and Performance, 26(3), 1209-1220. Download

What Does The Brain Do, Pt 2: The Fast Response System

2011-08-02T13:27:00.001+01:00

I want to continue thinking about the implications of the claim that the brain does not trade in representations. I'm not looking to defend this view here; we got into it a bit last time, I've talked about it here, and if you can't imagine what cognition without representation might look like, then you should read this post by Sabrina and then Radical Embodied Cognitive Science before worrying me with your lack of imagination. If you then feel like getting into it, Sabrina is tackling this topic in detail, beginning here.

If the brain isn't mentally representing, what is it doing? Last time, I got into the idea that the brain is part of a embodied cognitive system. It's in the middle of a rich information flow, with access to perceptual information about the world and ourselves, and it's a critical part of the action system, involved in our responses to that information. I talked about it as the fast response system in the set of inherent dynamical resources available for us to use to form task specific devices; I want to expand on that a little.

Fast Response System

The task specific device approach is primarily a methodology; instead of trying to study the perception-action system in general, you acknowledge that this system is too high dimensional, flexible and non-linear and instead focus on studying performance in well defined task spaces to identify how the system solves specific problems. It is also, however, how I think the system itself solves the problem. Right now, as I sit typing this, my perception-action systems are not trying to effect any of the other things I can, in principle, do. I am not, at this moment, actually capable of running. I have the ability to form a running device; but that, I think, is not the same thing. So the perception-action system resolves the 'degrees of freedom' problem by only trying to solve the problems at hand, using the resources available.

Such a system requires a) rich perceptual contact with the environment, and b) the ability to transition from one device to another, in response to changes in the environment. (This 'environment' can include internal states, such as goals and intentions - but I don't really know how to talk about these rigorously yet so I want to leave that for now.) The ability to switch the entire device between states takes time (inherent dynamical resources such as muscles and cardiovascular system all have inertia and particular response characteristics; change takes time, in other words). The nervous system doesn't have much in the way of inertia to overcome; it can, roughly, turn on a dime, and respond on a time-scale that will match the changing flow of information. It can therefore lead the formation of the next device.

One of the critical features of task specific devices is that they are, to a certain extent, functionally encapsulated. What I mean by this is that devices literally use some resources and not others; those being used are temporarily 'walled off' from those not being used, so that changes to irrelevant resources don't unnecessarily affect behaviour. This fact underpins the perturbation experimental method (which I've described here for action components, and here for perceptual components).

Which parts are 'walled off' into coherent systems changes over time as you go from being one device to another. You therefore need to be able to temporarily couple these components into systems with specific composition and organisation, and then dis- and re-assemble them into the next system. As I discussed last time, most of our inherent dynamical resources are quite stable over time (e.g. the link-tendon system; muscle composition), and the coupling between these systems is often physical and not malleable. Our entire body is innervated by the nervous system, however, and it's speed and flexibility is the mechanism that allows for the temporary coupling of disparate resources into stable devices.

Which device gets made?
There is, of course, the question of which device to make. Traditional motor programming type theories (such as Schmidt's 'generalised motor programme' approach) hold that you call up a set of commands for producing a class of movements, and, in response to cues from the environment, you set the parameters of that schema (how fast, how far, etc) to produce a specific movement. In essence, you must retrieve from memory a schema that contains what you've learned about that kind of movement, and only then adjust it. Sabrina discussed this topic in great detail here - this structuralist approach assumes that we must have stored a discrete object of some kind, which we can then tailor to suit the local demands. This approach is fraught with problems: how many of these can we store? How long does it take to find and retrieve the right one? How do we pick the right one? What sort of commands are stored in the programme, given how context sensitive movement is (moving to the same location in space can be achieved by a large number of different movements - the mapping is therefore many-to-one)? Given the more functionalist approach I am advocating, this is clearly not what I think is going on. So what do I think is happening?

I always think about batting in softball as an example of this. When you're going out to bat, you pick up the bat, and stand 'on deck' waiting your turn. In between pitches, we all swung our bats, warming up. When it was my turn, I'd get my feet planted and stand by the plate; I'd touch the far corner of the plate with the bat to get a sense of where the strike zone was and swing the bat a few more times from that stance. I'd also often let the first pitch go by, just to watch it. All of this was on the advice of the better players in our team.

What I was doing was generating information that was guiding the formation of a softball-hitting device. Swinging the bat generates information about it's inertial characteristics, which is the basis of dynamic touch (discussed at the end of this post on Chemero's book). I now have information about how the bat is changing my upper limb functionality, as well as information about what's required to move the bat (how long it takes, etc). Repeating this at the plate continues this flow of information in a more specific setting, driving the continued formation of the right kind of device. Watching the first pitch go by was also a way of sampling some task relevant information without yet trying to interact with it. My claim is that this is all happening in the moment; until I pick up a bat, I am not currently capable of swinging a bat. The job of the nervous system is to enable the formation of that device once I begin the process, and the specific device I form is the result of the structure of the flow of information and the kind of nervous system and body it's flowing into.

Conclusion
The job of the nervous system is not to store discrete representations with motor commands that produce a given response. Our skilled actions are exquisitely responsive to environmental demands and it's not clear that tweaking a discrete set of categorical movement types could possibly account for this. Instead, the nervous system is reshaping itself in response to the changing flow of information, and that new shape couples inherent and incidental dynamical resources into the task specific device you are capable of making that is related to the task at hand. The current form of the nervous system, like the current form of all our inherent dynamical resources, reflects how it is currently being used, and while there are, of course, identifiable changes you can relate to that use, those are not representations, and the only reason they are described as such is because representations are assumed; this kind of imaging result is not independent evidence for representation.

How good your device is, of course, depends on practice and learning, and what I haven't yet touched on is the other putative role for the brain - memory. Why was it that when I picked up a bat I began forming that device? Because that's what I had practiced. What was the consequence of that practice that remained so that I could use it again later? If I was a structuralist I would say 'procedural memory'; a thing, an object, that contained the results of that practice in some format. As a functionalist, I want to say 'perceptual learning'; through use, I have come to learn what that information means, and what it means has to do with swinging a bat to hit a softball. This is clearly not good enough, but it's next on my list to tackle, so hang tight :)

A Final Note: Part of the problem with which I'm struggling is finding ways of describing what the brain is up to in my terminology, without sounding like I'm merely redescribing what we already know. I think the brain is up to radically different things that representationalists, but often I'm talking about the same overall behavioural effect (like the softball bat) with no clear evidence that my different mechanism is actually the right one. Modern neuroscience has 'explained' many if not all of the kinds of phenomena I need to explain too, so I will eventually need to be able to distinguish these accounts empirically. But a) I can't rely much on the existing literature because it's framed incorrectly for me to use, b) I'm not a neuroscientist and won't be running more useful studies any time soon, and c) right now my goal is to start from the ground and work my way back up, exploring the consequences of my differing starting point (inspired by what I like most about Gibson, 1979). I'm not even close to completing this; these posts are just my thinking out loud, and so yes, I know I have no slam dunk. What I do need, though, is continuing feedback about the logic of my approach, and awkward questions about things you see as weaknesses continue to be very welcome!

A selection of problems with representation

2011-07-28T16:36:00.001+01:00

As Andrew alluded to in the comments to his last post, I'm cooking up some replies to various questions that have been raised. Being relatively time poor at the moment, I'm going to take these in small bits, rather than attempt a more comprehensive reply. The first thing I want to do is consider a few of the major issues, as I see them, with the notion of representations. As a cognitive psychologist, I spent many years endorsing a representational framework, but over time I have encountered enough problems/limitations with this approach that I consider the idea of representations to be seriously flawed. As a result, I've spent the past couple of years figuring out how to do cognitive psychology without them (but that's another post). What I want to do here are go through some of the things that have convinced me to abandon representations.

1) We don't know what we're talking about when we talk about representations

This criticism goes beyond the fact that there are multiple, competing definitions of representation. It goes beyond the fact that most cognitive psychologists are never asked to seriously wrestle with their own definition of representation, to consider its historical origins, or to consider whether there are alternative approaches to cognition. This criticism is more basic and is summed up nicely by Larry Barsalou: "We have no accounts of how propositional representations arise in the cognitive system, either innately or through experience. We haven't the faintest idea of how biological mechanisms could produce abstract propositions" (Barsalou, 1993, p. 173). Regarding this quote, Linda Smith says "When we cannot imagine how our basic ideas about cognition can possibly be realized, we ought to consider the possibility that they are wrong" (Smith & Jones, 1993, p. 181).

Granted, the field has moved on since the early '90s, but we still don't know much about the relationship between biology and cognition. And, I'm not talking about neuroimaging - interpretations of imaging data are ambiguous enough that they seem to reflect rather than challenge and inform our current understanding. I'm talking about how complex learning and memory are instantiated via networks of cells in a chemical bath.

2) We objectified the problem

Representations were initially invoked as a way to explain why some behavioural responses couldn't be completely predicted on the basis of the stimuli. Based on observations of the environment and our behaviour, it looks like something internal mediates our actions. Andrew has already discussed problems with the poverty of the stimulus argument, so I won't get into that aspect of the issue - there are other examples that are still potentially relevant even with a better understanding of perception/action. For instance, I might perform better on a math test when I am happy rather than sad. Or, I might take longer to respond to a question you ask if I am distracted by other thoughts than if I give you my full attention. Put simply, it is not the case that given a particular external environment I will always respond the same way (there's a good argument to make that objectively characterising an external environment doesn't make any sense because what matters is what goes into the system, not what exists out in the world, but that's a point for another day).

Early cognitive psychologists flagged up this problem and decided to spend some time figuring out what could be happening internally to account for our behaviour. This is a good idea. But, notice that this problem implies nothing at all about the form that internal processes/states/systems/whatever should take. It does not imply that the only way to account for behaviour is via internal discrete, computational representations. And yet, this is the dominant form of representationalism today (you might not know you're a discrete, computational representationalist, but chances are that if you believe in representations that you are one).

So, what do I mean that we've "objectified" the problem? I mean that cognitive psychologists chose to address the problem of internal mediating states with a noun - representation. Nouns are for doing things to. Acting upon. Messing with in some way. If our heads are full of things, of representations, then we must do things to them to get any use from them - you need processes to act on the things. You read a book, file a piece of paper, find a picture of a loved one. By objectifying representations, we treat them like objects to be created, filed, stored, sorted, retrieved, etc. These types of verbs populate much of cognitive psychology. But, creating, filing, storing, sorting, etc are only sensible things to do to discrete objects. If we do not have discrete representational objects in our head then many of the questions asked by modern cognitive psychologists need to be reformulated.

3) Discrete, stable concepts don't explain the most interesting and adaptive aspects of cognition

The traditional cognitive view is that we have object concepts (e.g., "dog") that contain relatively stable information about what we know about things. When these concepts are referred to in various contexts ("The dog chases the cat"; "The big, red dog") we think of basically the same thing. Of course, we're not actually thinking exactly the same thing - the stuff about dogs that comes to mind when you hear "The dog chases the cat" is not the same as the stuff that comes to mind when you hear "The big, red dog".

According to the traditional view, additional processes can modify the information about concepts that we have access to at any given time. Certain information is made "more salient" or is "weighted more heavily" based on goals or tasks. The idea is that we have core concepts, but might think of different aspects of them depending on context. Over time, the content of core concepts can be modified through learning and experience, but this modification is like editing a document - the document (representation) continues to exist, although some content may change over time.

Logically, contextual variability cannot be caused by stable conceptual representations. The variability must be caused by additional processes acting on the representation. As Smith and Jones (1993) note, this breaks cognition into "structure (stability) and process (variability)" making the task of cognitive psychology to figure out what stays the same across "different instances of a single cognitive act" (p. 182). Gelman and Medin (1993) point out that with this partition, the same set of data can be explained in multiple ways - simple representation / complex process, complex representation / simple process. This isn't good. But Smith and Jones (1993) additionally argue that stable concepts can't explain what's "smart" about cognition.

According to the traditional view, cognition is smart because it can represent concepts that reflect abstract and general information, which is most likely to be stable over repeated instances. Smith and Jones argue for a different type of smart cognition that prioritises flexibility and "fit" with a specific situation over stability. In the context of novel word interpretation, they say:

"If there is an abstract represented structure of some kind that sits behind the shape bias [children's tendency to generalise novel object terms according to object shape], it is the least interesting, least intelligent part, of the child's word-learning behavior. All the work that makes novel word interpretations smart is done by those processes that involve the specific objects and specific words at hand. It is these real-time, real-task processes that flexibly adjust attention to find the most likely referent of a specific utterance of some unknown word. Novel word interpretation is not smart because it is stable; novel word interpretation is smart because it is creatively adaptive. And for this kind of intelligence, perception--information about the here-and-now--always matters" (p. 184).

The nub of their argument is this: If representations are responsible for stability in cognition, but the really useful bits of cognition involve context-specificity and flexibility, then why are representations considered to be so important? This argument isn't a slam dunk against the notion of representations, but it does seriously question the focus of much of cognitive psychology. And, Smith and Jones don't deny that there is stability to cognition, but they do argue against the need for computational representations to account for this stability. Considering the enormous context-sensitivity of all purported cognitive processes, stable, discrete representations don't actually appear to be a very good way of modelling cognition.

So, what else could it be?

4) Representations were invoked to solve a particular problem (see point 2). But other solutions could have been proposed that solve the problem as well and probably better. Unfortunately, these solutions were never really explored and even now are only considered by a small minority of cognitive psychologists.

I've blogged in detail about this issue here, but here's a summary (based on Van Gelder, 1995).

The analogy: A major 18th century engineering problem was reconciling the oscillation of pistons with the rotation of flywheels. Driving a flywheel lets you generate rotative motion, rather than just pumping motion that results directly from piston. In other words, figuring out how to power a flywheel with pistons lets you power a wide range of machines. The trick is getting the flywheel to turn with uniform speed. Flywheels vary in response to the current steam pressure and to the overall engine workload. And, both of these factors, themselves, are variable. A throttle valve allows one to change the pressure of the steam, and therefore control the speed of the flywheel. But, this valve has to be adjusted by just the right amount at just the right time to keep the speed uniform.

The computational solution

One solution to controlling the valve requires something or someone to measure the state of the system at various points in time and adjust the valve by a certain amount in response to those measurements. If the steam pressure is x and the workload is y, then adjust the valve by z. The first characteristic of this type of solution is that it proceeds in stages. The first stage takes measurements. The second stage applies a rule based on those measurements. Because this solution relies on two stages, there is necessarily some time lag between measurement and correction. Depending on the duration of this lag, the correction might be inappropriate for the current state of the system. This type of solution necessitates an executive – someone or something to take account of the state of the system (e.g., “if x”) and then to carry out the appropriate action (e.g., “then y). It also necessitates measuring the difference between things. For instance, the only reason to adjust the valve is if the current speed differs from the speed a second ago.

This is the type of solution implemented in most cognitive models.

The dynamic solution

There is another, radically different way to solve the valve control problem - you can couple the opening of the valve to something that necessarily varies in response to steam pressure and workload in a way that results in constant flywheel speed. By ‘necessarily’ I mean that the physical properties of this thing respond to changes in steam pressure and workload in a particular way. Such a solution responds in one step and does not require measurement. Thus, there is no time lag problem or concomitant source of error. Nor does it require an executive. Hitching the valve control to the flywheel couples the thing you want to control (the valve opening) directly to the thing that embodies the relevant sources of variance (the flywheel speed). This beautiful solution is implemented in the Watts centrifugal governor, an 18th century piece of technology that still works brilliantly.

Consequences for cognition

The computational solution relies on discrete computational symbolic representations. It has to measure and represent flywheel speed, steam pressure, and workload using abstract symbols. Then, it has to apply operations to these symbols in order to calculate how to adjust the valve. This output is a representation that causes the appropriate adjustment to be made by an executive (computer or human worker). The second solution, the one that describes the actual centrifugal governor, is nonrepresentational. There are no discrete inputs and outputs, no computations performed on inputs and outputs, and no executive in charge.

While cognitive psychologists are often happy to admit that dynamical systems do a good job describing some systems like the centrifugal governor, they are hesitant to admit that dynamics might also characterise complex cognitive behaviour. So, here's a cognitive example:

According to prospect theory we compute the utility of various outcomes of a decision and select the one with the highest utility (Kahneman & Tversky, 1979). This theory clearly depends on discrete representations (i.e., of each option’s utility) and computation (i.e., calculating which option has the largest utility value). But, it is equally possible to describe decision making in terms of state space evolution in a dynamical system. For example, motivational oscillatory theory (MOT; cf. Townsend) describes oscillations resulting from satiation of persisting desires. We approach food when we’re hungry, but not when we’ve just eaten and are temporarily satiated. It’s possible to interpret this behaviour as a decision – when I’m hungry, I decide to eat. But, in Townsend's model there are no discrete states and no algorithmic processes effecting transformation on these states. There is just the evolution of the system over time. See Busmeyer and Townsend (1993) for a fully fleshed out dynamical model of decision making.

Whatever you might think of individual models, it is clear that dynamical systems are a legitimate alternative to the traditional computational approach to modelling cognition. It is also clear that these two approaches lead to very different assumptions about the nature of the underlying cognitive system. The computational approach leads us to think about mental objects (representations) that are manipulated and transformed via cognitive processes. The dynamic approach leads us to think about components in a cognitive system evolving over time and in response to the current context.

Summary

In my experience it's a waste of time trying to convince someone who endorses representational cognition that there are no representations. For one thing, representation is such a broad and ambiguous term that it is unclear what type of evidence could exist that unambiguously rules out representations. Sometimes people just want to keep using the term, so they'll apply it to aspects of dynamical systems (e.g., the location in state space of the MOT model "represents" hunger or satiety). Using representations in this manner doesn't contribute anything to the understanding of the system, and if representations aren't doing any work, then I can't see the point of continuing to invoke them. But this is besides the point. I really wouldn't expect a representationalist to be convinced by anything I've said.

So, what is the point of this post? The point is to demonstrate that representations were originally invoked to solve a particular problem. We currently have no mechanism to explain how representations could be instantiated in a messy biological system, so the representational framework does not have any special biological plausibility going for it. There are also alternative approaches to the problem of modelling mediating states (e.g., dynamical systems) that have several advantages over traditional computational models.

Given these points, I think that cognitive psychology would be vastly improved if cognitive psychologists 1) consider how they came to think that cognition was based on representations. Was it because they were taught from the beginning that cognition was representational? Or was it because they seriously investigated alternative approaches? Did they rule out these alternatives on the basis of data that favoured a representational approach? Or, did they rule them out on the basis of simply not being able to imagine cognition working without representations?

After asking myself these questions, I realised that I had unquestioningly adopted a representational framework because that is what I had been taught and that I had never actually considered alternative approaches. That didn't sit very well with me, and after spending a few years doing my homework I was convinced by the evidence that representations were not the solution to cognition.

Barsalou, L.W. (1993). Challenging assumptions abouy concepts. Cognitive Development, 8, 169-
180.

Busemeyer, J. R. & Townsend, J. T. (1993). Decision field theory: cognitive approach to decision making in an uncertain environment. Psychological Review, 100 (3), 432-459.

Gelman, S.A., & Me,din, D.L. (1993). What's so essential about essentialism? A different perspecfive
on the interaction of perception, language, and conceptual knowledge. Cognitive Development.
8. 157-167.

Kahneman, D. & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47, 263-291.

Smith, L. B. & Jones, S. S. (1993). Cognition without concepts. Cognitive Development, 8, 181-188.

Van Gelder, T. (1995). What might cognition be, if not computation. The Journal of Philosophy, 92 (7), 345-381.

There's More to Us Than Our Brains - So What Does The Brain Do?

2011-07-26T13:21:00.001+01:00

I'm not that interested in the brain.

It's hard to be this way in modern psychology. Cognitive neuroscience is where it's at, and I think I come off as a bit of a Luddite when I try to convince people fMRI is a bit of a waste of time. Not caring much about the brain is certainly a sociological reason why ecological psychology doesn't get taken very seriously; we're just the crazy people who don't think there are mental representations, based on some work from the 50s-70s. Surely modern imaging has shown us the activity of mental representations? Clearly, the brain is the source of all behavior! Popular science writing on psychology is all cognitive and representational; most of the psychology blogging I come across is neuroscientific. What else could it be?

I've certainly spent a lot of time waving the flag against the infiltration of neuro-talk into places it doesn't yet belong; but to be honest, as I get older, I've begun to worry that I'm trying to be 'fair and balanced' in the sense Fox News is fair and balanced: relentlessly playing up one side to offset a perceived imbalance elsewhere. What I actually want to do is be actually fair and balanced: I want my own discussions about these issues to be internally balanced and coherent, giving credit where credit is actually due. I want to start teasing apart a few issues I've conflated over the years, so that my strong concerns about the relevance of fMRI and cognitive neuroscience work stop getting swallowed up in a general dismissal of the brain's role in our lives. The brain is clearly interesting, but it's not representing, and if not that, what is it doing?

This post is therefore a first swing at integrating a lot of the things I've been blogging about for a while and doing so in a way that leaves a sensible role for the brain. I'm going to need some neuroscientists to talk to, though; I'd appreciate it if people could spread the word on this a little, because there are just some things I want to go a few rounds on with people who know what they're talking about.

What is the brain doing?
It's clearly the case that the brain is up to something; it consumes something like 20% of our body's energy, and our bodies are thoroughly innervated by projections to and from the nervous system. So what, precisely, is it up to?

Incorrect Answer: Mentally representing the world

Whatever it's up to, the brain is not representing the world. My main reason for thinking this is that this argument is firmly based in the assumption of poverty of stimulus, which is entirely incorrect. My very first post on this blog was about this; modern cognitive psychology is committed to this assumption, and everything it thinks the brain is up to stems from this assumption. In actual fact, we are embedded in a rich information flow to which we have a great deal of direct access, so representation just isn't really required any more.

A note about what I think representations are for

Cognitive psychology is dedicated to the idea that the input to the system is insufficient to support the kinds of behaviour we see. Mental representations are mediating states which have content, and this content is used to supplement the input so as to make it good enough to produce the behaviour we actually observe. But they are only required if the input is impoverished, and the point of the ecological approach is that the input is rich and specific to the environment. If this is the case, then the brain doesn't need to be mentally representing anything.

There are a couple of reasons why people think the stimulus is poor. The tipping point for the cognitive revolution in the 1960s was Chomsky's review of Skinner's Verbal Behaviour, which claimed to show that the input for language simply wasn't structured enough to support a learning story. In perception, the poverty of stimulus assumption is rooted in studying the anatomy of the eye. The retina seems to only be capable to producing an upside-down, horribly noisy and pixelated image, and so clearly this input needs serious enriching in order to support the visual experience we are all familiar with. Gibson spent some time detailing why this is ridiculous; vision is not derived from static images and the retinal image is an invention of scientists, an analytic tool, not a fact of perception. Interestingly, while Chomsky's ideas are alive and well in linguistics, psychology is getting increasingly interested in the richness of the statistical structure of linguistic input and starting to tell interesting learning stories.

So what is the brain doing then?

The brain is not some isolated, abstract executive. The idea of the brain in a vat is incoherent and a distraction; real brains are utterly integrated with a wide variety of other systems as well as the world around it, via perception. Brains have an important job to do, but they just aren't the only player in the game and there's a lot of things they simply don't need to do because some other system takes care of it. (Done carefully, this is the point of embodied, enactivist theories of cognition). To work out what the brain needs to do, one thing we have to do is understand what everything else is up to. We need job descriptions for the brain and the bodies they're embedded in.

Here's what I believe to be true:

Our behaviour emerges over time as we respond to the flow of information in our environment. We are not general purpose systems - we are, instead, at any given moment, one kind of task specific device (and importantly, not another kind). The kind of device we currently are is a function of what we've been up to recently; the specifics of the device reflect the nature of multiple subsystems and how these respond to the flow of information we are currently embedded within (more on this in a moment; bear with me).

This information flow is surprisingly stable; we are not adrift in a 'blooming, buzzing confusion' (to use the regularly misused William James quote). One consequence of this stability is that we can rely on it to do a lot of work for us, and there is plenty of evidence that we do just this. My favourite example is change-blindness, a phenomenon in which people can be easily made to not notice dramatic changes in their environment by removing the information specifying that a change has occurred. The early versions of these studies involved 2-frame animations of, say an aeroplane. In one frame, the engine was present under the wing, and in the other frame it's missing. You play this animation over and over and ask people whether anything is changing, and people take ages to find it, if ever. There's a critical trick, however: the animation has to be filtered to remove what's called a transient signal: the abrupt change in pixel values creates a blip, if you like, in the signal. If you filter this out, but still have two very different frames, then people simply often do not see the change. There are more recent and ambitious demonstrations, including some great real life examples (see this video, for example) but they all also require that something covers up the information that something has changed (a camera cut, or an intervening event).

Change blindness tells us that we do not store a representation of a scene; instead, we simply perceive the flow of information about the scene and respond. We don't see the change because the information specifying that a change has occurred has been removed - no information, no basis for noticing the difference. We float along in a flow of information and our behaviour emerges as we interact with that flow.

We aren't passive blank slates, however. At any given moment in time we are very specific measurement devices, sensitive to some information variables and not others and capable of responding in some ways and not others. We spend our days being one device, then another, then another, in response to changes in the information flow and in accordance with our capacities. The information flow alters as our location in space and time changes, and is specific to the current environment; our capacities reflect the current state of multiple subsystems, of which the brain is only one.

The kinds of devices we are

If I measure my height with a ruler calibrated in inch units, I get the number 67 out. If I do it with a ruler calibrated in centimetre units, I get 170. It's the same amount of space, but my two measurements have produced different results because they are calibrated differently. If I want to measure area and only have a straight ruler, I have to make several independent measurements and combine them in a computation; if I have a polar planimeter, I can measure the area directly, no computation required. The 'simple' unit depends on the device and can, actually, be quite higher order (see this old post for what I still think is an excellent explanation using right angled triangles).

The moral of those two points is this: perception is an act of measurement, and the result of an act of measurement depends on the device doing the measuring. The perception-action approach claims we perceptually measure the world using measurement devices calibrated in action-relevant units; the output is therefore action-scaled without any further manipulation required and can be plugged directly into the action system in question.

The trick here is that the actions we are attempting to perform are constantly changing, and thus the device we measure the world with needs to be regularly re-calibrated to different scales. Proffit's work on distance perception clearly demonstrates that we perceive the 'same' distance differently depending on whether we intend to traverse it by walking or throwing. This is what I mean when I say we move through the day being one device, then another, then another - as the required actions change, so must our perceptual measurement systems.

Calibration & Subsystems

We are, in turns out, very flexible devices. Healthy adults can calibrate and recalibrate swiftly and efficiently, although it takes years to be fully competent at this (hence motor development is a long, slow process). Recalibration does take time (we are physical systems with inertia and delays) but then persists for a while without active maintenance (for roughly the same reason); it can, of course, be maintained more robustly if the required information remains present.

We build these devices out of the locally available resource dynamics; the combination of the task-related incidental dynamics and our internally available inherent dynamics (definitions and details here). We couple ourselves to incidental dynamical resources via perception (e.g. tool use); we use perception in an identical fashion for our inherent dynamics (there's no privileged access, no peeking behind the curtain - we only know what perception tells us), but these are obviously special in one sense, namely that we take them from one task to the next.

Our inherent dynamical resources come from a variety of subsystems. Each contributes different elements to a task-specific perception-action device. Most critically, I think, these systems work at different time-scales to provide a careful balance between stability and flexibility, and this is where the brain starts to enter back into things for me. Bingham (1988) describes the following basic divisions in the inherent dynamics. These subsystems are all coupled to one another, and their basic behaviours are non-linear, as are the couplings.

The link-segment system: our skeleton is hooked together in a very specific way, and this arrangement enables some behaviour and rules some other behaviours out. This arrangement doesn't fundamentally change over time (barring injury, although the composition of the bones remodels in response to load-bearing exercise over long time scales), and thus it serves as a very stable basis for the other systems. The role of this system is to provide a physical substrate for the transmission of the forces involved in moving and interacting with the world, and it places useful constraints on the kind of motions that are possible.
The musculotendon system: the size, shape and composition of this system is partly constrained by genetics, but is of course highly responsive to use. The time course is days and weeks, and so this system is highly stable over short periods of time but flexible and responsive to demand over longer time scales. The role of this system is to generate the forces required to move the link-segment system; it's organised in ways to solve many of the 'degrees of freedom' problems inherent in controlling a complex system.
The circulatory system: This system is highly responsive to current events: it only takes 10s of seconds for heart rate to accommodate current energy demands, and veins and arteries change size and shape in response. The role of this system is to deliver energy to the musculotendon system so it can generate the forces required to move the link-segment system.
The respiratory system: This is another highly responsive system which adapts to current requirements on very short time-scales. It's role is to provide oxygen and remove carbon dioxide, to enable the continuing metabolic processes powering the muscles.
Nutritional systems: this operates over longer time scales, and it's role is to provide the nutrients and energy required for the above systems.

Finally, the nervous system. The brain and it's innervation of the body via the peripheral nervous system seems to me to be the fast response system in all this. The nervous system responds on microsecond time scales, and is in constant flux. It's connected to everything, and is, I think, the primary medium for the informational coupling between the various subsystems (not the only one; a lot of these systems have direct physical contact, for instance).

The brain is dynamically stable, possibly (probably?) ideally edge-of-chaos stable. It exhibits a great deal of structural stability (visual cortex is in the same place in everyone), but even this stability is being actively maintained by the stability of the informational flow. Change that flow (for example, have someone pick up a tool, or provide information that you actually have three hands) and that structure smoothly alters in response. The reason visual cortex is where it is and organised the way it is, therefore, is because that's what a nervous system looks like when it is exposed to that kind of informational flow. Change the flow, change the organisation (as happens when people become blind, for instance: that cortex is 'colonised' by neighbouring functionality because it's structure as visual cortex is no longer being maintained).

Besides this actively maintained stability, the brain also ebbs and flows in fast response to changes in that flow. I remember taking Olaf Sporn's neuroscience class and watching microscope video of neurons extending and retracting axons on millisecond timescales; neurons aren't static at all, they are alive and frankly they are busy as fuck. The brain is alive and on the go and what it looks like at any given moment is a function of the information flow it is currently embedded in.

Trying to get a rigorous handle on this kind of dynamic behaviour is a daunting task. What makes me think it's possible is the fact that people are trying seriously to do it. I mentioned Olaf Sporns; he's a neuroscience professor at IU and he's about the only neuroscientist I've ever seen in person grapple seriously with the dynamic nature of the brain. He gave a talk once to the cognitive science reading group when I was there, and I came away thinking that 'yes, that's it, that's the way to do business'. I'm slowly reading his excellent book Networks of the Brain, and the purpose of that book is to connect neuroscientists with the modern science of network analysis so they can give up on 'biologically plausible neural networks' (you know, the little PDP toys we all got impressed with in the 80s and 90s) and actually try to cope somehow with the multi-scale structure of the brain.

(One of the ideas I like a lot already is that of topological neighbours. Topology is a flavour of geometry that describes the set of possible transformations and relations possible when you relax the rules as far as you can; no need to preserve metric distances, for example. Networks can contain regions which are not physically next to one another but that are informationally coupled so that they are effectively part of the same network This informational coupling can then ebb and flow over time, so a given bit of cortex can be involved in multiple systems doing different things at different times. Of course, not all combinations are possible; there are anatomical limits on who is connected to whom and by how many steps. It's not clear to me that fMRI based neuroscience can cope with this kind of data, and wouldn't know it was seeing it if it saw it.)

So what is the brain doing? With our clearer understanding of what's going into the total system, we can start making sensible guesses.

I think the nervous system is the fast switching system that enables us to functionally wall off resource dynamics to form task specific devices, a walling off which lasts only as long as it's supported by the flow of information. I think the nervous system routes action scaled information to the systems that can use it. In network geometry terms, I think the nervous system is a high dimensional shape that constantly changes configuration in response to information, preserving some aspects and transforming others. I think the notion of invariance-over-transformation is a geometric tool that applies equally to this network as to the optic flow Gibson applied it too; there is flux, but critically there is invariance and that invariance is information. I also think these are all very different from what most people seem to think goes on in the brain.

I don't think the brain is performing all the computations supposedly required to move a limb, given a) many of those 'computations' are actually solved by the architecture of, say, the hand, and b) not enough people seem to know about the way the nervous system actually moves limbs, namely via equilibrium point control. I don't think the nervous system is the place to look to explain stability in behaviour; the globally stable environment we live in and have perceptual contact with solves that problem.

Now all I need to do is make these slightly mad sounding ideas make more sense and hold together better. Annoying questions welcome :)

Lissajous feedback and coordination stability

2011-07-19T11:47:00.002+01:00

Understanding the perceptual information you provide people in a task is a critical element of the perception-action analysis. Last time I talked about the new form of coordination feedback I developed to allow us to train coordinated rhythmic movements without perturbing the task dynamic. Prior to this, the most common form of augmented feedback was the Lissajous plot - these are the result of plotting the displacements of two harmonic oscillators against one another, and the unique shape associated with each relative phase can be used as a template on the screen. People can then try to move so as to make a dot trace that shape.

Lissajous plots (have a play with them in this Excel file) are transformed feedback, because they take a coordinated movement and represent it on the screen as the motion of a single dot. This type of feedback has been used extensively to train people to perform novel coordinations, but until recently no-one had thought to investigate the consequences of transforming the information about relative phase. Kovacs, Buchanan and Shea have recently begun doing exactly this, and, in line with the perception-action approach developed by Bingham and pushed at every opportunity by myself, these authors have found that Lissajous plots completely alter the nature of the task, with serious consequences for the studies that rely on it.

Performing 90° with Lissajous feedback

Kovacs et al (2009a) investigated whether people could produce 90° using Lissajous feedback without extensive practice. It typically requires a least a couple of days practice before people can do this; however, Kovacs et al correctly identified that the Lissajous format display really should be easier to use than that, because it consolidates the coordination information into a readily detected signal. They note that most studies pace frequency with an auditory metronome, and also often allow vision of the limbs. The latter had recently been shown to be a problem (Shea et al, 2008).

They compared performance at 90° with Lissajous feedback, with and without a metronome and with no vision of the hands. Without the metronome, the visual feedback display stabilised performance over 10 30s trials (about 5 mins). With the metronome, performance was terrible, both variable and inaccurate. Both groups suffered when the feedback template was removed for two trials at the end, suggesting they both depended on it and hadn't actually learned to produce 90°, just how to track the feedback.

Summary: Lissajous feedback can quickly stabilise performance of 90°, but the commonly used metronome is a major distractor (to add to 'vision of the limbs' from the previous study).

Performing other novel coordinations

Kovacs et al (2009b) then tested whether this effect generalised to other novel coordinations (0°-180° in 30° increments). One group performed using Lissajous feedback, no metronome and no vision of the arms. The second group performed using a visual metronome, with vision of the arms (cf the 'scanning' sessions developed by Zanone & Kelso). There were three practice blocks from 0° to 180° and back again, followed by the test block in the same format. All data are from the last block, and the feedback template was always on.

The visual metronome group produced the typical result; 0° and 180° were produced most accurately and stably, with 180° less stable than 0°. All other coordinations were performed very inaccurately and with high variability. In contrast, the Lissjous group performed all 7 coordinations with high accuracy, and while the non-0° conditions were more variable than 0°, they were still more stable than under the metronome. With about 20 minutes practice and no distractions, the Lissajous group were able to reliably produce any coordination.

Summary: Lissajous feedback, in the absence of distractors, can quickly allow stable movement at any relative phase with only slight increases in variability.

Amplitude Variations

One of the ways to reduce the stability of coordinated rhythmic movements is to make people move their limbs at different amplitudes. You typically see amplitude assimilation where the two amplitudes converge on some intermediate value and resisting that makes the coordination harder to maintain. These errors often also lead to problems maintaining the target relative phase, and, on top of all of this, there are asymmetries in all these effects due to handedness.

Kovacs & Shea (2010) tested whether Lissajous feedback helps people moving at different amplitudes at either 0°, 90° or 180°. With Lissajous feedback, movement accuracy was good, although variability looked more typical - higher at 90° than 180° which was higher than 0°. However, movements were more accurate and more stable with than without Lissjous feedback. Lissajous feedback also reduced amplitude assimilation, but only when the dominant right hand was performing the smaller amplitude.

Summary: These results are less clear cut, but they do demonstrate that Lissajous feedback does help to stabilise the coordination, and was sometimes able to help stabilise the amplitude control too. This partial effect may have something to do with the fact that Lissajous feedback represents amplitude differences statically, in terms of the radius of the template figure. We think (from an upcoming paper) that amplitude is actually perceived in terms of energy, the radius of the movement on the phase plane. The trouble people had using the Lissajous plot to control amplitude may have something to do with this. This is just something that has come up very recently, so as yet I don't have a more formal story. However, my hunch is that, again, these results support the broader attempts to keep perception firmly in the story.

Overall summary

In general, Lissajous feedback stabilises all kinds of otherwise difficult movements. It does this by removing the need to visually perceive a coordination; instead, all you need to do is perceive a single dot which represents that coordination. The perceptual task no longer varies in difficulty across relative phase, and the stable perception of the target allows stable behaviour. This is great evidence in favour of the perception-action approach and (with our coordination feedback data) for the role of the perception of coordination information as the primary source of the HKB phenomena.

Kovacs et al are mostly on board with that analysis; they cite our work and talk about it sensibly, which makes a refreshing change. There's one aspect of their analysis I don't think quite works, though. They note that the dynamic pattern approach suggests the bistability of 0° and 180° is the result of attractor dynamics. They then suggest, however, that, on the basis of their results, the bistability actually seems to be a side effect of attentional confounds (metronomes, vision of the limbs). This I don't agree with; I think our coordination displays, which still produce bistability without metronomes or vision of the limbs, confirm that the bistability emerges from the overall perception-action dynamic, of which information is a key part. These attentional issues clearly matter; but I don't think they are the ultimate source of the key structure. Still, some excellent results which I intend to engage with empirically over the next year; I'm confident I'll be able to extend and support these data.

References

Kovacs, A., Buchanan, J., & Shea, C. (2009a). Bimanual 1:1 with 90° continuous relative phase: difficult or easy! Experimental Brain Research, 193 (1), 129-136 DOI: 10.1007/s00221-008-1676-2

Kovacs, A., Buchanan, J., & Shea, C. (2009b). Using scanning trials to assess intrinsic coordination dynamics Neuroscience Letters, 455 (3), 162-167 DOI: 10.1016/j.neulet.2009.02.046

Kovacs, A., & Shea, C. (2010). Amplitude differences, spatial assimilation, and integrated feedback in bimanual coordination Experimental Brain Research, 202 (2), 519-525 DOI: 10.1007/s00221-009-2154-1

Shea, C.H., Buchanan, J.J., & Kovacs, A.J. (2008). Cooperation between the limbs is better than we thought. Journal of Sport & Exercise Psychology, 30, S128.

Visual feedback for training novel coordinations

2011-07-12T11:40:00.000+01:00

The key feature of coordinated rhythmic movements is that not all coordinations are stable. Most other rhythms can be learned, however, which is why we can have jazz drumming. People have been training participants to perform novel coordinations (especially 90°, the least stable rhythm without training) for years now, and have been asking all the standard learning questions - how long does learning take? Does it transfer to other coordinations?

The first real studies on learning were by Kelso and Zanone (Kelso & Zanone, 2002; Zanone & Kelso, 1992a, b, 1997). I briefly reviewed the results of these studies here, which have lead to to the dynamic pattern hypothesis. This account describes stable states as attractors in a state space defined by relative phase as the order parameter, and learning is the creation of a new attractor centred on the target novel phase. This account ran into problems quite quickly but is still alive and kicking in a modified form; stability is the governing principle now, and from this perspective the feedback displays used for training don't matter so long as they support stable action.

However, from our perception-action standpoint, the feedback displays matter a lot, because these are what's providing the perceptual information about the coordinated movement. Early learning studies all used some kind of transformed feedback, which we could never use because it altered the overall perception-action dynamic. In order to look at action learning directly, we needed a new form of feedback.

So I invented one.

Forms of Transformed Visual Feedback

1. Visual Metronomes

Figure 1. Visual metronome

The first form of feedback was the visual metronome, used by the Kelso & Zanone studies, and is still used by Zanone. All it is is a display of two dots who alternate their position at some frequency and some relative phase to each other. Figure 1 shows two dots alternating at 180°. Participants are instructed to (for example) flex and extend their index fingers to match the dots; the left dot for the left finger, etc. The net result is that they move at the required relative phase and frequency.

This effectively reduces the coordination task to two tracking tasks; people no longer have to actively produce a coordination, they simply have to track two signals and they end up producing a coordination. We used this trick in a previous paper (Wilson, Bingham & Craig, 2003) to have people produce coordinations they normally couldn't so they could make proprioceptive judgements about the movements. Coordination is not the same as two tracking tasks, however, and the key phenomena emerge from the coordination requirement. The upshot is that with this feedback, people aren't coordinating anything and thus it's no use to us. In addition, there is no motion in these displays and thus relative direction is not defined.

2. Lissajous Figures

Figure 2. Lissajous plot

A second form of transformed feedback is the Lissajous figure. These are the result of plotting the displacement of two harmonic oscillators against one another. The resulting plot is specific to the relative phase; for movements at the same frequency, 0° produces a line of slope 1, intercept 0, 180° is a line with slope -1, and 90° is a circle. Other relative phases are ellipses of varying eccentricity (this figure on Wikipedia lays out the space, and this Excel file I made let's you play with the parameters to generate the resulting shape). You can also specify more complex frequency relations with more complex plots.

You can display a template shape on a screen and ask people to move so as to make a single dot follow the shape. But this is also transformed feedback; there is motion but only one dot (so nothing to be coordinated and no relative direction), and again the task is reduced to tracing a shape. Work from Charles Shea's lab has explored this in detail and shown that Lissajous plots effectively make all coordinations equally easy and don't actually readily promote much learning; I'll review these in more detail next time.

Coordination feedback

We needed a form of feedback that didn't perturb the task dynamic we're interested in studying, i.e. the one described by the model. I therefore used the normal display of two dots, and simply changed the colour to green when the person was moving at the target relative phase, +/- some error range. Colour has precisely no effect on coordination stability (Mechsner & Knoblich, 2004) and is also not part of the task dynamic described by the model. It can therefore serve as a task-neutral 'hot-cold' signal to participants, who must actively produce a coordinated movement to get the feedback.

We therefore described and tested this feedback method, to make sure it works (Wilson et al, 2010; download). There are two ways to implement the feedback (described in the Appendix) but the net result is the same: the dots change colour to signal when you are within a given range of the target relative phase. We trained participants to move at 90° by starting the feedback error bandwidth quite wide and then making it harder to get the feedback over time by narrowing the window. We also gave a control group exactly the same amount of practice with these displays, but no feedback. The results are in Figure 3.

Figure 3. Movement stability data for the learning and control groups.

The result was unambiguous - participants improved their coordination stability at 90° to levels equivalent to 180°, but only when trained with the feedback. The control group failed to improve at all (due to the fact that prior to training, 90° is simply not clearly perceived). The feedback cued people when they were moving correctly, and in the Post-Training session (with the feedback turned off) they showed that this enabled them to learn how to actively produce a novel coordination. Note also how much more robust this improvement is than the gains with perceptual training - this is a perception-action task and while perception is a critical part, the movement itself matters too.

Summary

This method completes our methodological toolbox. The perception-action approach meant we had to invent numerous new methods to study coordination, including visual judgements, proprioceptive judgements, perceptual training, perturbation displays, and finally action training. This feedback method was only motivated by the perception-action approach; no one else thought the feedback mattered, despite increasing evidence to the contrary. We are now explicitly comparing performance with this feedback and the transformed versions, looking to see what, if anything, people learn about coordination using the other methods.

References

Kelso, J. A. S., & Zanone, P. G. (2002). Coordination dynamics of learning and transfer across different effector systems. Journal of Experimental Psychology: Human Perception and Performance, 28(4), 776-797.

Mechsner, F., & Knoblich, G. (2004). Do muscles matter for coordinated action? Journal of Experimental Psychology: Human Perception and Performance, 30(3), 490–503.

Wilson, A. D., Bingham, G. P. & Craig, J. C. (2003). Proprioceptive perception of phase variability. Journal of Experimental Psychology: Human Perception and Performance, 29(6), 1179-1190. Download

Wilson, A., Snapp-Childs, W., Coats, R., & Bingham, G. (2010). Learning a coordinated rhythmic movement with task-appropriate coordination feedback Experimental Brain Research, 205 (4), 513-520 DOI: 10.1007/s00221-010-2388-y Download

Zanone, P. G., & Kelso, J. A. S. (1992a). Evolution of behavioral attractors with learning: Nonequilibrium phase transitions. Journal of Experimental Psychology: Human Perception and Performance, 18(2), 403-421.

Zanone, P. G., & Kelso, J. A. S. (1992b). Learning and transfer as dynamical paradigms for behavioral change. In Tutorials in Motor Behavior II, Advances in Psychology (Stelmach, G., & Requin, J., Eds.). Amsterdam: North-Holland.

Zanone, P. G., & Kelso, J. A. S. (1997). Coordination dynamics of learning and transfer: Collective and component levels. Journal of Experimental Psychology: Human Perception and Performance, 23(5), 1454-1480.

Rates of learning and the dynamic pattern approach

2011-07-06T14:14:00.000+01:00

One of the interesting features of coordinated rhythmic movement is that people start out with a particular pattern to their performance - there is pre-existing structure to our attempts to coordinate these movements. This structure affects our ability to learn new coordinations, and the pattern of the effects reveals a lot about the cause of this pre-existing structure.

However, the literature is split into two incompatible accounts of learning, and trying to fix this is part of my ongoing interest in this task. The first account is the dynamic pattern approach, which was pioneered by JAS Kelso, and championed by modelling (Gregor Schöner) and behavioural studies (Pier Zanone). I'm more interested in the latter aspect, because it's the motivation for the former. I've already reviewed how this account fails, but it's still alive and well thanks to some creative history, and needs to be tackled again. The second account, which I prefer, is the perception-action account (Bingham) which developed from empirical work on visual and proprioceptive perception as well as action measures, and embodied in a model.

We haven't explicitly tackled the rate of learning issue, although we will and there is already support for our account in the literature (Wenderoth et al, 2002). But it comes up regularly in the dynamic pattern behavioural work, so it's time to work out what's going on in their data.

Predictions of the dynamic pattern approach

Zanone & Kelso (1994) published a book chapter summarising the work so far and laying out the dynamic pattern approach. In it, they made a specific prediction:

...learning rate should vary inversely with the stability of the closest intrinsic attractor to the required pattern.
Zanone & Kelso, 1994, pp 482

In other words, learning a novel coordination closer to 0° should be harder than learning one close to 180°, because the stronger attractor at 0° will interfere more with the learning process. This is not what occurs; in fact, Fontaine et al (1997) and Wenderoth et al (2002) found the exact opposite pattern and suggested that this reflects people's ability to perceive the required information - the region around 0° is more readily discriminated, which is the root cause of it being so stable.

It seems straight-forward: the dynamical pattern hypothesis makes a clear and straight forward prediction and two papers have shown the prediction is incorrect, plus the perception-action account can readily explain the result. Case closed, surely.

Well, no. The dynamic pattern approach is still motivating empirical work. Zanone now frames the discussion of learning in terms of stability: learning a novel coordination near a stable state (e.g. 0°) is more likely to persist in memory because stable is better than unstable. Zanone now regularly replicates the result that learning, say, 45° is easier than learning 135° but claims it supports the predictions of the dynamic pattern approach, based in stability.

This move happened, as far as I can tell, in a paper by Kostrubiec & Zanone (2002). The authors first claim (incorrectly) that the dynamic pattern approach predicts that competition between stable states and novel coordinations being learned varies with distance such that competition is weakest when the distance is small. This is the exact opposite of the actual prediction, quoted above and which they cite as if it supports the claim. It also makes no sense: a stable state is, in dynamical systems terms, an attractor, and the ability of an attractor to influence a system's behaviour can only decrease with distance. This misrepresentation of their past work is a real concern, although later papers acknowledge the original prediction (and then claim the dynamic pattern approach still explains what actually happens).

As an initial attempt to attack this empirically, Kostrubiec & Zanone (2002) attempted to measure learning, memory and interference using three different tasks during training:

Learning: this involved attempting to move at the target relative phase, paced by a visual metronome (either 90°, 135° or 158°, which are 90°, 45° and 22° away from 180°, respectively). This assessed the effect of distance from an attractor (although it ignores the asymmetry in attractor strength between 0° and 180°).
Memory: this task was called synchronisation-continuation, and involved moving for 10s with the metronome on then 20s with it off; the latter part tests memory for the pattern being learned.
Interference: this involved moving at a currently stable relative phase other than the one being learned. If you were currently learning 135°, you were prompted (with an 8s demo) to move at, say, 90°. I think this is the case; the methods are entirely unclear at this point. I believe the idea was to measure how hard it was to produce a recently learned phase in the middle of learning another one.

The results of this messy fishing trip were, predictably, an incoherent mess.

Effects of the various manipulations on movement accuracy (absolute error, AE) were confounded with variation in movement stability (standard deviation of the AE). This is a problem with these studies, because movement stability is not independent of mean performance, and you can produce a stable movement by not accurately producing the difficult condition you are being asked to make. We fixed this by moving to a 'time-on-task' measure of movement stability; I'll blog the details of that measure soon as it will come in handy later.
Training continued until people's movement stability with the metronome on reached a criterion. However, it's clear from the data that in the absence of the metronome performance was terrible, so it's not at all clear anyone has learned anything. While pre- and post-training performance across a wide range of relative phases was measured, only the baseline data are reported.
Rate of learning was longest for 135°, so it was not linearly related to distance from 180°. This makes little sense, and possibly comes from the fact that people had to learn the three different relative phases in order.

The interpretation of the data is all framed within the incorrect description of the dynamic pattern predictions, and the model fit presented in Figure 2 is, I'm pretty certain, a data fitting exercise, not a prediction of the model. This is only an educated guess because the model fit procedure is not described, but the HKB model doesn't really have any way to produce that figure without data.

Summary

There's more results to go over in future posts, but while the methods get tighter and the reporting of past work less shoddy, the pattern is going to be the same as here: with no actual mechanism at work, there is nothing constraining the explanation of the data and everything, no matter how it works out, fits the model. 'Stability' doesn't explain anything, because it can be used to explain everything. Without a mechanism, these experiments are just data-fitting exercises; my goal is, of course, to explain the data with reference to the model.

References

Fontaine, R., Lee, T., & Swinnen, S. (1997). Learning a new bimanual coordination pattern: Reciprocal influences of intrinsic and to-be-learned patterns. Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale, 51 (1), 1-9 DOI: 10.1037/1196-1961.51.1.1

Kostrubiec, V., & Zanone, P. (2002). Memory dynamics: distance between the new task and existing behavioural patterns affects learning and interference in bimanual coordination in humans. Neuroscience Letters, 331 (3), 193-197 DOI: 10.1016/S0304-3940(02)00878-9

Wenderoth N, Bock O, & Krohn R (2002). Learning a new bimanual coordination pattern is influenced by existing attractors. Motor control, 6 (2), 166-82 PMID: 12122225

Zanone, P.G., & Kelso, J.A.S. (1994). The coordination dynamics of learning: Theoretical structure and experimental agenda. In S.P. Swinnen, H. Heuer, J. Massion, & P. Casaer (Eds.), Interlimb coordination: Neural, dynamical, and cognitive constraints (pp. 461-490). San Diego, CA: Academic Press.

Chemero (2009) Chapter 9 - The Metaphysics of Radical Embodiment

2011-06-21T13:03:00.000+01:00

The final chapter of RECS tackles the metaphysical implications of the radical stance. Gibson was a staunch realist, but there are some odd elements to entities like affordances that, to certain minds, sound like idealism or antirealism of some kind. Realism is, essentially, the claim that there is a world independent of our experience of it, and that we can have basically accurate knowledge of that external world. In modern times this reality has been equated with the description given by physics. Affordances don't belong to physics, however; whether relations or dispositions, they are, at heart, facts which span the organism and the environment. This sounds wrong to a lot of ears (as Ken's comments on that post readily show!). Chemero therefore devotes the final chapter to defending the claim that RECS can be realist; this matters, because people tend not to like idealism in their science these days, and it's going to be a standard philosophical objection to the RECS programme if not addressed.

Realism

The problem is this: embodied cognition stories describe organisms who only perceive certain things which are very personal to themselves (e.g. affordances) and who essentially operate within a world defined by their own sensori-motor abilities. This sounds like idealism, the idea that we all simply inhabit our own little worlds and these aren't connected to anything external to the organism. This is clearly an issue.

Many non-representational stories simply reject the realism/idealism dichotomy - Varela et al (1991), for example, claim that replacing representation with embodied action simply changes the game so that they can dodge the problem. Andy Clark thinks this move doesn't work, and part of his plan for a non-radical embodied cognitive science involves action-oriented representations, who's content is firmly grounded in an objective world. Chemero, however, thinks Clark runs into the same basic problem, because even an action-oriented representation must be tied to specific sensori-motor capacities and runs the risk of locking an individual up inside their own abilities. This just makes Clark's representations walk into essentially the same idealist trap as he thinks non-representational accounts suffer from.

One potential way out of this mess, Chemero suggests, is plasticity. The above story, that organisms are locked into an understanding of the world dictated by their sensori-motor capacities and are therefore idealists, falls apart once you allow for neural (or other) plasticity. Plasticity enables a system to adapt and change in response to changes in it's environment; allowing this requires allowing a path from the world to what the organism knows about the world. Problem solved.

Entity Realism
So if plasticity allows RECS to embrace realism, what flavour of realism should it embrace? Philosophy has spent a long time delineating an ever increasing list of realisms; Chemero advocates entity realism (Hacking, 1982, 1983). According to this flavour, you are entitled to claim something is real (i.e. exists independent of you) and that you can have accurate knowledge of that thing when you are able to use the thing to achieve some scientific goal. Hacking's example is the electron; scientists are justified in believing electrons to be real because they can use them to do things (e.g. investigate other particles). This makes the electron part of the lab just like a glass beaker is part of the lab, and so it is at least as real as that beaker is.

So entity realism says something is real if it can be used as part of your scientific practice. Chemero simply extends this idea to embrace other practices, such as action. Affordances can therefore be real because they can be used as part of the successful practice of acting in the world. Chemero suggests this maps on nicely to the whole RECS approach to things. And while there are, of course, philosophers who don't think entity realism is tenable, the basic approach is robust enough to at least serve as a sensible first draft. Actually this is mostly just a stance, still, and as such is a bit weak, but Chemero acknowledges the limitations here.

Chemero then describes three strands of scientific research which assume affordances and use these to investigate other aspects of perception and action; this is another way to demonstrate how entity realism lets affordances being perfectly real.

Affordances and events: Chemero and colleagues have been able to manipulate the presence and absence of gap-crossing affordances and systematically vary the behaviour of participants (Chemero et al, 2003).
Affordances and distance perception: Chemero cites the work of Dennis Proffit and colleagues (which I have mention here and will come back to in the next few weeks). Proffitt et al vary the affordance and task structure to investigate the perception of distance, i.e. how it is scaled.
Affordances and perception of tools: Casler & Keleman (2005, 2007) have manipulated the affordances of tools to investigate how children attribute functions to objects.

Two Non-Problems: Qualia & Mental Causation
Chemero then explains how two classic problems in philosophy of mind have no traction in a radical embodied cognitive science.

The first problem is mental causation: how can something like cognition cause physical things to happen? How does causation cross levels of explanation, e.g. from the action of neurotransmitters to the movement of my laptop keys? Chemero claims that a RECS based in dynamical systems doesn't have the levels problem. Take the HKB model (or at least, take the model as described by Chemero earlier; I don't think this specific example quite does the work Chemero needs, although I buy the broader point). Relative phase is an order parameter which is composed of lower level elements and yet also controls their behaviour. The kind of level-ignoring causation in dynamical systems seems to be the right kind of causation for RECS.

The second is the problem of conscious experience. Computational systems can have content, and so, for computational cognitive science, my mind and my laptop can both have states which represent some state of affairs. But only I can have the conscious experience of having this knowledge; my laptop lacks qualia.This problem is a side effect of a computational solution to the problem of how we get up to things, and thus RECS doesn't have this problem. RECS is, in fact, all about accounting for our experience of the world as meaningful - meaning is not a separate element which must be somehow included in the mix, it's part of the act of perception.

To many, this claim simply attempts to dodge a bullet with an extensive pedigree - the problem of qualia and how to get conscious experience into our lives has been around for a long time. But it's actually a positive hypothesis, a claim which RECS makes on the basis of everything that's come before in this book.

Summary
And that's it! Let me end this extended trawl through this book with a brief summary of what I think this book is good for.

As you may recall, I have a few detailed problems with the specific flavour of ecological psychology Chemero lays out:

I think affordances need to be dispositions, and not relations, and I think all the reasons Chemero cites for the relational story are entirely accounted for by the fact that the act of perception is relational. Chemero & Turvey have banged this around some in a technical paper I frankly didn't understand, but the practical upshot is that the two accounts, relational and dispositional, essentially amount to the same thing. OK, fine - I'll treat affordances as dispositions and not stay too worried about this fight.
I think that the situation semantics expansion of the idea of information is interesting, but worrying. I think it loses too many important features (the symmetry principle, for instance) and again, I wasn't entirely convinced by the motivations. That said, Sabrina and I have been going round and round on this question, and there are times when I can see how relaxing the law based requirement might help. I'll keep thinking about this.

I also think Chemero's use of the HKB model is a swing and a miss. This isn't his fault - his read is indeed the way the model gets talked about in the literature. But it's an incorrect analysis, and Sabrina and I are working on a paper this summer laying out the case for a better style of modelling based on Bingham's model of coordination. Any suggestions for good journals to send this to gratefully received :)

So I have some quibbles with some details. But Chemero achieves his primary goal - laying out a positive, productive approach to cognitive science that doesn't rely on the crutch of mental representation. This is very, very important: most cognitive scientists assume that such a thing is entirely impossible even in principle, and it is without a doubt the single most important reason the work of James J Gibson has been sidelined for so long. Given that the ecological approach is a highly successful theory and the root of a strong empirical programme responsible for introducing dynamical systems to psychology, this side lining by many scientists is a real problem. It's also a problem with consequences: the notion of affordances has been readily embraced by cognitive science, but untethered by the complementary notion of information it has lost all it's meaning and has been reduced to one of those terms people throw in as if they know what it means.

So my quibbles aside, I like this book and it does an important job. Now it's time to spend a little time doing radical embodied cognitive science, and trying to draw the literature into the broader conversation about the way we study the mind and behaviour.

References
Casler, K., and D. Kelemen (2005). Young children’s rapid learning about artifacts. Developmental Science, 8, 472-480. Download

Casler, K., and D. Kelemen (2007). Reasoning about artifacts at 24 months: The developing teleo functional stance. Cognition, 103, 120-130. Download

Chemero, A., C. Klein, and W. Cordeiro (2003). Events as changes in the layout of affordances. Ecological Psychology, 15, 19-28. DOI

Hacking, I. (1982). Experimentation and scientific realism. Philosophical Topics, 13, 71-87.

Hacking, I. (1983). Representing and Intervening. Cambridge: Cambridge University Press.