Is Indirect Perception Plausible?

There are two basic ways perception might work to let us experience the world in behaviourally relevant ways. Direct perception is the idea that perception only requires two components; the environment and the organism. Indirect perception is the idea that perception requires at least three components; the environment, the organism, and at least one other component that mediates between the organism and the environment. Over the last few posts, I've been working through the specifics of the ecological approach to making direct perception plausible, because this is a question I often get (usually in the form of 'I don't see how this could work in this case'). Regardless of whether or not it's correct, we can show that we have all the pieces needed to make direct perception work in principle, and the empirical programme is about seeing if it works in practice. What about indirect perception?

I asked this question on Twitter, and one interesting thing I noticed was just how little sense the question seems to make to people these days. Responses fell into roughly two categories: 'I don't see how we can do with it in this case', and 'brains do stuff, so...', neither of which answer the question. Even if some form of indirect perception is required in those cases (which is, of course, still up for grabs) we're still owed an account of how this might work, at least in principle and then later in practice. 

People used to know this. The most recent indirect perception hypothesis is that the key mediator is a mental representation, understood as a computational, information processing system implementing some form of inference that combines sensory data and information stored in memory to create a model of the world that represents the system's best guess about what is out there and how to behave successfully with respect to what's out there. This hypothesis didn't come out of nowhere; the development of the computer and the theory of information that allows them to work turned out to provide the pieces required to create a formal account of representations that stood a chance of living up to the challenge of explaining perception. Cognitive scientists therefore leaned heavily into the details of these pieces as they worked very hard, from the late 1950s on, to make indirect perception implemented this way plausible. 

The exact details of the process have, of course, changed and evolved with empirical data and developments in computational theory. For example, while all the accounts have to do inference that combine sources of information into a best guess, there are a variety of ways of doing inference, some better than others. Probably the best way to do inference is via Bayesian methods, and so most modern theories propose that indirect perception combines sources of information this way so as to be optimal. 

Before these inferential methods can even be brought into play, however, there remain two related and big unanswered challenges that need to be addressed. The first one is the grounding problem; how do representations get the content they need so as to combine sources of information in a way that works? It's all very well describing the inferential process of the fully formed system, but how do you build one in the first place? The second is the 'which representation?' problem; of all the different sources of information the system has to combine, how does it know which information to bring together for a given task? These reflect a circular problem indirect theories create for themselves. If perception is not good enough to be direct, and thus requires representational support, where do those representations come from? In order for a theory of indirect perception to be plausible, these must be addressed (analogous to how in order for a theory of direct perception to be plausible, questions like 'can the physical world present itself in behaviourally relevant ways?' had to be addressed). 

I am not going to address these challenges to indirect theories, because it isn't my job. But they are legitimate questions that people have mostly stopped asking. Debates about the form and content of representations were prominent and explicit right up until the end of the 1990s, and then it all just seemed to stop. Interface theory, for all it's problems, at least got back into the fight and tackled the grounding problem (unsuccessfully, I've argued, but it was a solid swing and at least Hoffman recognised he owed us an account). Mark Bickhard's work is probably the only currently active research programme explicitly working out the details, but I don't know many scientists who even know who he is, and a lot of his work is about mapping out the rules of living up to the challenge, versus actually solving the problem. 

Until these foundational issues are addressed and answered, whether indirect perception is plausible remains unclear, and no matter how sophisticated your inferential machinery is (looking at you, free energy principle) it can't help until you explain how it came to be organised that way in the first place. Even if the ecological theory of direct perception doesn't hold up, representational theories of indirect perception are not viable options if they cannot be shown to be plausible. 

Lecture 7: Empiricism and the Man in the Inner Room (Turvey, 2019, Lectures on Perception)

The last two Lectures have laid out the mechanical, Cartesian analysis of the problems of perception. Every flavour of this analysis, right up to an including modern representational accounts, rests on 'loans of intelligence', in which important work that's required to make the system work is simply given to the system. For example, Descartes proposes a set of cognitive axioms to ground the inferential processes required to implement knowledge, and makes these innate (specifically, given by God). This won't do - if we want to develop a naturalised theory of perception, we need an account of where every part of the system comes from. 

The Cartesian programme is a form of rationalism, roughly the claim that knowledge is based in reason rather than experience. This chapter reviews the opposing camp of empiricism, roughly the claim that knowledge is based in experience rather than reason. Turvey reviews the approaches of Locke, Berkeley, and Hume, with respect to how they assert a mind can come to know about the world via experience. They end up running into very similar problems as Descartes et al, because of their continued commitment to the mechanistic hypothesis. We are going to have to reject both of these as we progress.

Ecological Information Is a Perceptual Mapping That Tracks Evolutionary Fitness

Interface theory in cartoon form. Thanks to Louise
Barrett for reminding me this exists :)

In my last post I was thinking out loud about some ecological lines of attack on interface theory (Hoffman et al, 2015). The first line of attack (Hoffman et al mischaracterise Gibson) fell over eventually; they don't quite go at it right, but using ecological information does fit their definition of a naive realist perceptual strategy ( 'a perceptual strategy for which X [perceptual experience] ⊂ W [the world] and P [the perceptual mapping] is an isomorphism on this subset that preserves all structures on W'; pg 1483). The second line of attack (everything they say about veridicality vs fitness applies only to inferential, constructivist theories of perception and Gibson's not playing that game) is true but not that interesting or convincing to anyone with established views.

Thanks to chats on Twitter (thanks Greg!) and applying the basic move of 'those aren't working but IT is still weird, what's next?', my new line of attack relates to a result from their simulations.

Does Interface Theory Have Consequences for the Ecological Approach?

I've been working on a commentary about interface theory (Hoffman, Singh & Prakash, 2015) which I have previously blogged about here. I'm still interested because it is, in part, a fairly direct shot at the ecological approach and I'm always keen to break those if I can. My piece stalled out, though, so I thought I'd spend some time here thinking out loud about the argument that stalled and another critique that came up as I re-read the paper.

To unbury the lede I just finished writing: the primary thing Hoffman et al get wrong about Gibson is that they think he wanted his theory to produce veridical perceptions, not simply adaptive ones. Gibson actually wanted adaptive perceptions, but found a way in which they were also veridical. This emphasis matters; Gibson does not stand or fall on issues of veridicality. In addition, every one of Hoffman et al's big swings apply only to inferential, constructivist theories of perception; Gibson is immune on these grounds as well. All Hoffman et al have done is redraw the terrain inferential theories have to traverse, and it will be interesting to see if anyone takes the bait. But the major argument simply remains, is perception inferential or ecological, and may the best data set win.

The Interface Theory of Perception - The View from Ecological Psychology

Psychonomic Bulletin & Review has released, with much fanfare and a hashtag, an article called 'The Interface Theory of Perception' (Hoffman, Singh & Prakash, 2015). From the website description:

In a nutshell, interface theory postulates that our perception operates like a species-specific desktop: We perceive the world in representations that do not represent the “truth” about the world as it actually is, but that are useful “icons” which represent fitness-relevant information about the world. To illustrate, imagine a world in which red and green berries are nutritious but blue and yellow berries make you sick. Will your perceptual system differentiate red from green and blue from yellow? According to interface theory, the answer is no—the organism will have evolved to differentiate between only two colors, namely gred and byellue.

The meat of the paper is a series of evolutionary simulations that pit various perceptual strategies against one another. These strategies vary in how veridical they are, and the key result is that interface strategies, in which perception codes things in a way that bears no resemblance to the world, wins every time. We do not perceive the world as it really is. 

This seems to go against people like Gibson, who argue that perception is of a real world and real properties of things, like affordances. These simulations seem to show that 'realist' perceptual strategies are evolutionarily unsustainable. 

The devil, as always, is in the details, and having read the paper I am now pretty sure that Gibson is quite safe, and that information offers a path out of the weirdness Hoffman conjures.