The Interface Theory of Perception - The View from Ecological Psychology

Psychonomic Bulletin & Review has released, with much fanfare and a hashtag, an article called 'The Interface Theory of Perception' (Hoffman, Singh & Prakash, 2015). From the website description:

In a nutshell, interface theory postulates that our perception operates like a species-specific desktop: We perceive the world in representations that do not represent the “truth” about the world as it actually is, but that are useful “icons” which represent fitness-relevant information about the world. To illustrate, imagine a world in which red and green berries are nutritious but blue and yellow berries make you sick. Will your perceptual system differentiate red from green and blue from yellow? According to interface theory, the answer is no—the organism will have evolved to differentiate between only two colors, namely gred and byellue.

The meat of the paper is a series of evolutionary simulations that pit various perceptual strategies against one another. These strategies vary in how veridical they are, and the key result is that interface strategies, in which perception codes things in a way that bears no resemblance to the world, wins every time. We do not perceive the world as it really is. 

This seems to go against people like Gibson, who argue that perception is of a real world and real properties of things, like affordances. These simulations seem to show that 'realist' perceptual strategies are evolutionarily unsustainable. 

The devil, as always, is in the details, and having read the paper I am now pretty sure that Gibson is quite safe, and that information offers a path out of the weirdness Hoffman conjures.

The Apparently Astonishing Insight of the Simulations
Let's walk through their main example. 

Figure 1. Depictions of two perceptual strategies

Say you have a resource, and this varies in quantity from 0-100 (x axis on the plots). You have a payoff function (the Gaussian) such that having some of the resource is better than having all or none (think about water; none and you die of thirst, all and you drown, some and you are good). You have 4 possible perceptual states; red, yellow, green and blue. How do you go about mapping perceptual states to resource quantity so that perception gets you the best payoff?

The left hand picture shows a realist perceptual strategy, which preserves the structure of the world (the resource) in the perceptual mapping. The four colours get mapped uniquely to states of the world. Perceiving red means the resource quantity is 25% or less; perceiving yellow means 25-50%; and so on. (This is what Hoffman thinks someone like Gibson is claiming; it's not, but more later.)

The right hand picture shows an interface strategy. The four colours get mapped uniquely to regions of the payoff curve. Perceiving red means the payoff is minimal, perceiving blue means the payoff is maximal, and so on. The interface perceptual strategy does not preserve the structure of the world (the resource) in the perceptual mapping; it is a 'fiction' that supports high payoffs. 

Hoffman et al pit these guys and some others against each other in evolutionary simulations. The critical outcome is that 

strict interface strategies that are tuned to fitness routinely drive naive realist and critical realist strategies to extinction

and they say 

The key insight from these evolutionary games is this: Natural selection tunes perception to payoffs, not truth.

Not So Astonishing
There's a problem: the simulations are rigged against truth. 

The rigging is simple; the resource quantity varies in a linear, monotonic way. Payoff does not. Any mapping that preserves world structure does so by preserving the monotonic pattern; such a mapping therefore struggles to fit the nonmonotonic payoff function. The interface mappings are not so bound, and so fit the payoff functions better. And the simulations of evolution track fitness by payoff. So the evolution equations naturally favour mappings that can follow payoff well. (Hoffman et al note truth does ok when it tracks payoff, which fits this analysis.) 

This means they need an external motivation for payoff being a non-monotonic function of the world. They simply assert this to be true ('monotonic functions are a (unbiased) measure zero subset of the possible payoff functions' which I think just means monotonic functions are such a tiny subset of functions that set size is effectively empty; set theory be crazy. Thanks qmaqdk and mnxmnkmnd for Twitter help!). They then say 'well homeostasis makes non-monotonic more common' (think back to the water example underlying the Gaussian payoff). This is a little more useful but only a little; it assumes the only thing we perceive is 'amount of stuff' for example. So all this undercuts the simulations, I think.

Gibson: Not So Screwed After All

Hoffman et al class Gibson as a realist mapping, because the phrase 'direct perception' sounds like it should be describing a realist mapping. In his reply to various commentaries, Hoffman also notes

The evolutionary simulations we report were based on an exhaustive and mathematically precise classification of all possible definitions of “veridical perception”. None of them can compete successfully against interface perceptions that are sufficiently tuned to fitness. Thus we did not pick some particularly easy straw men and test them; we tested the whole gamut of possibilities.

This is not true, however. Ecological psychology is not in this mix. This is partly because Hoffman et al don't really understand Gibson (their 'critique' at the end, for example, is laughably naive). It's partly because phrases like 'direct perception' are a little misleading (more on this in a future post). And it's partly because, like everyone, Hoffman et al focus on affordances and pay no attention to information (weirdly, if they understood information, they might have classed Gibson as an interface theory; it meets the definition!).

But it's mostly because Gibson's ecological approach is a genuinely radical, revolutionary theory of perception that simply isn't playing the same game as the other players. As a result, Hoffman et al don't have him in their games either, and their results have no consequences for the ecological approach.

1) Interface theory never tested ecological information
Every single one of Hoffman's mappings are representations that are the result of an inferential process. He is committed to perception-as-inference and only tests this idea. In such a scheme, a perceptual mapping is something an organism has to build, and so the idea that there might be a bunch of ways to build such a mapping is a sensible corollary. Hoffman et al have indeed tested all the possibilities within this scheme and found inferential representations have to be interfaces.

Ecological psychology is different. It proposes that instead of an organism having to invent a perceptual mapping, it's job is to discover the one and only one that exists; the mapping of dynamics to kinematic structures in energy arrays via the laws of ecological optics (for vision). The organism then needs to learn to use those structures as information to coordinate and control behavior, via specification or convention. If this is the case, then none of Hoffman et al's analysis applies - there is no competition between mappings because there are no options. As far as perceptual science is concerned, the main argument is still just about information vs inference and nothing has changed here from this paper.

As a side note, this means that, to the extent that there are consequences for theory from this paper, they are entirely restricted to inferential theories. These consequences might be as bad as a necessary commitment to a crazy ontology, or they might be as simple as a move away from making representations a veridical map of the world-as-described-by-physics. But if this theory is mind-blowing, it's only blowing inferential minds. 

2) Interface theory does not rule information out as an option
There's still the possibility that interface theory still rules ecological psychology out because it's a theory about how we make contact with a world that interface theory says isn't really there (Greg Hickok's been trying this one out on me via Twitter). Maybe 'dynamics' is just part of our interface, so there's nothing to map into kinematics. 

But, to steal a phrase from Ed Baggs, ecological psychology does not stand or fall depending on the latest fads in the physics department. We do not perceive 'the world as measured by physics', a point Gibson made clear a long time ago. It is part of the job of an empirical science of perception to characterise 'the world as measured by perceptual systems', and when we do that job we end up with properties like affordances, which no theory of physics in it's right mind would ever invent.

I see it like this. 

• No one wants to be a solipsist (Hoffman notes in his replies) so that means that there is stuff independent of us and our experience of it. 
• Physics measures that stuff in certain ways and classifies the stuff into things like particles and forces. This is just a theoretical description and might be a poor way to classify the stuff, although so far, so good for the Standard Model. 
• Ecological psychology argues that perception measures that stuff in different ways and classifies that stuff into things like affordances and events. Again, the specifics of this theory might be wrong, but we'll find out once we've scienced it a bunch more. 

But at the end of the day, if there is stuff, there is the possibility for some of that stuff to map itself onto other parts of the stuff just by interacting with it, and if an organism begins to detect the latter there is the possibility of using it to perceive the former. There are two ways to do this; using that latter as information, or using it as the basis for inference. But if there is stuff, and we all assume there is, then both of these options remain on the table, and finding out which one works is still the job of perceptual science. 

Summary
Interface theory argues that the structure of perceptual experience cannot map onto the 'true' structure of the world in any straight-forward way. The experience we end up with is an interface, a mapping of convenience that allows us to interact with our environments in ways that promote evolutionary fitness rather than knowledge of the truth. 

This only works if we get to make our own interface, and the ecological approach argues that this is not the way it goes. Instead, evolution builds sensors that interact with the one and only existing 'interface', kinematic structures in energy arrays created by the lawful interaction of the world dynamics and that energy. And, despite it's best efforts, interface theory neither refutes nor rules out this as a legitimate option. 

Interface theory might indeed by 'mind blowing' but only for inferential minds. If there are troubling consequences, they only affect representational psychology. Otherwise nothing has changed - the argument that matters is still just 'information vs inference', and so let's do the science and may the best data win.