Lecture 17: Pattern Recognition and Representation Bearers (Turvey, 2019, Lectures on Perception)

Last Lecture Turvey reviewed the basics of the computational approach, and highlighted again how it is just the latest iteration of the Cartesian programme. In this Lecture, he explores the specific topic of pattern recognition, which has been a major topic in the computational approach and exemplifies many of the major problems. These problems primarily boil down to systems requiring loans of intelligence to even come close to working.

Lecture 16: The Computational-Representational Perspective: Preliminaries (Turvey, 2019, Lectures on Perception)

The next four Lectures are specifically focused on the modern form of the Cartesian programme - the computational-representational approach. It's important to spend some time here, because this is the ecological approach's current opposition, and because the fact it is just yet another Cartesian programme matters, and is at the heart of most of our objections to it. 

In Lecture 6 we learned about the 3 grades of sense. First, there is reflex. The second is limited awareness of secondary qualities. The third is full mental awareness of what it all means. The proposed solution (Hobbes) for building the third grade out of the other two is the manipulation of symbols to do inference. In modern times, this is implemented as computation, and implemented in a representation.

Is Indirect Perception Plausible?

There are two basic ways perception might work to let us experience the world in behaviourally relevant ways. Direct perception is the idea that perception only requires two components; the environment and the organism. Indirect perception is the idea that perception requires at least three components; the environment, the organism, and at least one other component that mediates between the organism and the environment. Over the last few posts, I've been working through the specifics of the ecological approach to making direct perception plausible, because this is a question I often get (usually in the form of 'I don't see how this could work in this case'). Regardless of whether or not it's correct, we can show that we have all the pieces needed to make direct perception work in principle, and the empirical programme is about seeing if it works in practice. What about indirect perception?

I asked this question on Twitter, and one interesting thing I noticed was just how little sense the question seems to make to people these days. Responses fell into roughly two categories: 'I don't see how we can do with it in this case', and 'brains do stuff, so...', neither of which answer the question. Even if some form of indirect perception is required in those cases (which is, of course, still up for grabs) we're still owed an account of how this might work, at least in principle and then later in practice. 

People used to know this. The most recent indirect perception hypothesis is that the key mediator is a mental representation, understood as a computational, information processing system implementing some form of inference that combines sensory data and information stored in memory to create a model of the world that represents the system's best guess about what is out there and how to behave successfully with respect to what's out there. This hypothesis didn't come out of nowhere; the development of the computer and the theory of information that allows them to work turned out to provide the pieces required to create a formal account of representations that stood a chance of living up to the challenge of explaining perception. Cognitive scientists therefore leaned heavily into the details of these pieces as they worked very hard, from the late 1950s on, to make indirect perception implemented this way plausible. 

The exact details of the process have, of course, changed and evolved with empirical data and developments in computational theory. For example, while all the accounts have to do inference that combine sources of information into a best guess, there are a variety of ways of doing inference, some better than others. Probably the best way to do inference is via Bayesian methods, and so most modern theories propose that indirect perception combines sources of information this way so as to be optimal. 

Before these inferential methods can even be brought into play, however, there remain two related and big unanswered challenges that need to be addressed. The first one is the grounding problem; how do representations get the content they need so as to combine sources of information in a way that works? It's all very well describing the inferential process of the fully formed system, but how do you build one in the first place? The second is the 'which representation?' problem; of all the different sources of information the system has to combine, how does it know which information to bring together for a given task? These reflect a circular problem indirect theories create for themselves. If perception is not good enough to be direct, and thus requires representational support, where do those representations come from? In order for a theory of indirect perception to be plausible, these must be addressed (analogous to how in order for a theory of direct perception to be plausible, questions like 'can the physical world present itself in behaviourally relevant ways?' had to be addressed). 

I am not going to address these challenges to indirect theories, because it isn't my job. But they are legitimate questions that people have mostly stopped asking. Debates about the form and content of representations were prominent and explicit right up until the end of the 1990s, and then it all just seemed to stop. Interface theory, for all it's problems, at least got back into the fight and tackled the grounding problem (unsuccessfully, I've argued, but it was a solid swing and at least Hoffman recognised he owed us an account). Mark Bickhard's work is probably the only currently active research programme explicitly working out the details, but I don't know many scientists who even know who he is, and a lot of his work is about mapping out the rules of living up to the challenge, versus actually solving the problem. 

Until these foundational issues are addressed and answered, whether indirect perception is plausible remains unclear, and no matter how sophisticated your inferential machinery is (looking at you, free energy principle) it can't help until you explain how it came to be organised that way in the first place. Even if the ecological theory of direct perception doesn't hold up, representational theories of indirect perception are not viable options if they cannot be shown to be plausible. 

How Worried Are You By The Symbol Grounding Problem?

Imagine you're a mental representation. You are a computational symbol system, and your job is to contain knowledge that is about the world and that can help your organism interact with that world (Newell, 1980). The 'aboutness' thing is the most important part of you - you are an intentional system, which means you have content that is meaningful.

So where did your content come from? (I'd like to know your thoughts, so please help by answering the questions at the end!)

Oh crap. Re-thinking van Gelder (A purple peril)

I have this problem where I like pretty much everything William Bechtel writes except when it pertains to cognitive science. It's annoying because, even when I disagree with him, I think he's worth taking seriously. This was on my mind when I started reading is 1998 paper, "Representations and cognitive explanations: Assessing the dynamicist's challenge in cognitive science."

In this paper, Bechtel critiques van Gelder's construal of the Watt's governor as an exemplar of how cognition could function without representations. I have always liked van Gelder's argument and wrote about it years ago in very favorable terms. I wouldn't say that I don't like it anymore, but Bechtel may have convinced me that one of van Gelder's central claims - that representations don't feature in explanations of the Watt governor - is incorrect. Does this mean that I now think that cognition requires internal representations? No! But, the language Bechtel uses to assess van Gelder's arguments sounded eerily familiar and I realized that the criteria used to label something as a representation are met by...ecological information.

Thinking about representations in relation to mechanisms

As Chemero (2011) observed, there are two ways to think about debates in science. We can either debate about the actual facts of the matter in the world or we can debate about the best way to explain the world. Some debates aren't amenable to the first type of debate, because there is no evidence that can definitively rule out one of the options. The only debate we can really have about representations is in terms of their role in explanations of behaviour. This is different than how I thought about representations a few years back when I wanted to argue that invoking representations was inherently a bad idea. Now I think we need to consider the utility or representations as part of explanations for psychological phenomena. In this post, I will argue that the concept of representations is not helpful in developing a particular class of explanation - ontic mechanistic explanations (described below). This is the first of two posts on this idea. In the next post I will attempt to explicitly compare cognitive and ecological approaches to behaviour in terms of how well they set us up to identify real parts and operations.

The Interface Theory of Perception - The View from Ecological Psychology

Psychonomic Bulletin & Review has released, with much fanfare and a hashtag, an article called 'The Interface Theory of Perception' (Hoffman, Singh & Prakash, 2015). From the website description:

In a nutshell, interface theory postulates that our perception operates like a species-specific desktop: We perceive the world in representations that do not represent the “truth” about the world as it actually is, but that are useful “icons” which represent fitness-relevant information about the world. To illustrate, imagine a world in which red and green berries are nutritious but blue and yellow berries make you sick. Will your perceptual system differentiate red from green and blue from yellow? According to interface theory, the answer is no—the organism will have evolved to differentiate between only two colors, namely gred and byellue.

The meat of the paper is a series of evolutionary simulations that pit various perceptual strategies against one another. These strategies vary in how veridical they are, and the key result is that interface strategies, in which perception codes things in a way that bears no resemblance to the world, wins every time. We do not perceive the world as it really is. 

This seems to go against people like Gibson, who argue that perception is of a real world and real properties of things, like affordances. These simulations seem to show that 'realist' perceptual strategies are evolutionarily unsustainable. 

The devil, as always, is in the details, and having read the paper I am now pretty sure that Gibson is quite safe, and that information offers a path out of the weirdness Hoffman conjures.

Why don't giraffes fall over more often?

One of the major reasons psychologists think we have mental representations is to overcome delays in the nervous system. Information has to come from sensors such as the retina and travel the distance to the visual cortex. This takes time. That information must then be processed and combined with other information to generate adaptive responses. This takes time. Delays in a control system that relies on feedback are a big problem for stability. The more time it takes for feedback about how well you are doing to reach the brain, the less relevant that feedback information is - it's no longer about what you're doing right now. This makes it difficult to make sensible error corrections and it ups the chances that something will go badly wrong. A lot of people therefore claim that the only solution is prediction, and there's a strong research industry investigating how the nervous system predicts so well.

Grounded vs. embodied cognition: a (hopefully uncontentious) note on terminology

Our Frontiers paper made the case that embodied cognition is, by definition, a fairly radical affair. We argue 

...if perception-action couplings and resources distributed over brain, body, and environment are substantial participants in cognition, then the need for the specific objects and processes of standard cognitive psychology (concepts, internally represented competence, and knowledge) goes away...

We argue that this is compulsory; as soon as you allow the perceived environment to play any kind of critical role in cognition, it's game on for what Shapiro calls 'replacement style' embodied cognition. This is why we don't think that we're just at one extreme end of a continuum of embodiment research; we think the rest of the field is making a category mistake. Chapter 2 of Chemero (2009) does an excellent job of laying out the history here; representations come from the structuralist school of thought, embodiment from the functionalist school. They are, quite literally, two different kinds of approach and mixing them is just an error.

I need to talk about this other stuff, though, and I'm tired of calling it 'not embodied cognition'. For one thing, the critical tone gets in the way of the argument. Shapiro calls it 'the conceptualisation hypothesis' and while this is basically accurate, it's a slightly non-intuitive technical term I'd have to explain all the time. So I want to be slightly cheeky and rename that other work, while picking a name they will hopefully not mind. This work, I think, is really grounded cognition (as per Barsalou, 2008) and that's how I'll talk about it from here on in.

What's the difference between perception and conception?

As Andrew has been tackling a new job description for the brain (part 1 and part 2), several comments have been made that suggest that his approach (and the ecological stance in general) might be fine for perception/action, but not for other types of tasks/behaviours. Later on in this post I also think about how we might be able to distinguish between association and direct access to meaning, which is another idea that has been coming up repeatedly in the comments (see here).

In this post I want to think about what makes perception different from conception (Andrew reviews William James' views on this distinction here). I will argue that both occur as networks of evolving neural activity (with perception, this network extends to the environment and the body), but these networks have different properties because they are driven by signals of differential stability. I do not think it is accurate to think of perception as something that happens in V1, for example, and conception as something that happens in SFG. If parts of the brain reverberate in a system along with parts of the brain that are directly interfacing with an energy array in the environment, it seems correct to refer to that whole system as perception. In contrast, if a network of activity appears to be relatively encapsulated with respect to external energy arrays, it seems correct to refer to that system as an act of conception. For now I will leave aside the problem of how conceptual networks develop, but I hope to return to this later.

What Does The Brain Do, Pt 2: The Fast Response System

I want to continue thinking about the implications of the claim that the brain does not trade in representations. I'm not looking to defend this view here; we got into it a bit last time, I've talked about it here, and if you can't imagine what cognition without representation might look like, then you should read this post by Sabrina and then Radical Embodied Cognitive Science before worrying me with your lack of imagination. If you then feel like getting into it, Sabrina is tackling this topic in detail, beginning here.

If the brain isn't mentally representing, what is it doing? Last time, I got into the idea that the brain is part of a embodied cognitive system. It's in the middle of a rich information flow, with access to perceptual information about the world and ourselves, and it's a critical part of the action system, involved in our responses to that information. I talked about it as the fast response system in the set of inherent dynamical resources available for us to use to form task specific devices; I want to expand on that a little.

A selection of problems with representation

As Andrew alluded to in the comments to his last post, I'm cooking up some replies to various questions that have been raised. Being relatively time poor at the moment, I'm going to take these in small bits, rather than attempt a more comprehensive reply. The first thing I want to do is consider a few of the major issues, as I see them, with the notion of representations. As a cognitive psychologist, I spent many years endorsing a representational framework, but over time I have encountered enough problems/limitations with this approach that I consider the idea of representations to be seriously flawed. As a result, I've spent the past couple of years figuring out how to do cognitive psychology without them (but that's another post). What I want to do here are go through some of the things that have convinced me to abandon representations.

1) We don't know what we're talking about when we talk about representations

This criticism goes beyond the fact that there are multiple, competing definitions of representation. It goes beyond the fact that most cognitive psychologists are never asked to seriously wrestle with their own definition of representation, to consider its historical origins, or to consider whether there are alternative approaches to cognition. This criticism is more basic and is summed up nicely by Larry Barsalou: "We have no accounts of how propositional representations arise in the cognitive system, either innately or through experience. We haven't the faintest idea of how biological mechanisms could produce abstract propositions" (Barsalou, 1993, p. 173). Regarding this quote, Linda Smith says "When we cannot imagine how our basic ideas about cognition can possibly be realized, we ought to consider the possibility that they are wrong" (Smith & Jones, 1993, p. 181).

Chemero (2009), Chapter 4: The Dynamical Stance

The problem with the theory of representation Chemero settles on in the previous chapter, as we shall see, is that it makes representation unavoidable. If there are representations, then RECS fails to get off the ground (remember, the radical bit is anti-representationalism). In this chapter, Chemero identifies a path past this problem (the 'dynamical stance'), explains why it works, but then concludes with a new problem, that of how to guide future discovery.

Chemero (2009), Chapter 3: Theories of Representation

The radical part of 'radical embodied cognitive science' is anti-representationalism. Simply put, the claim is that the brain does not internally represent states of the world in any way, nor are these the basis for our experience. It's radical because to a typical cognitive scientist, the mere suggestion is ludicrous: like suggesting you do biology without genes or particle physics without Large Hadron Colliders. But this indignation is hiding a dirty little secret: mental representations are generally just assumed to exist, because of assumptions about the poverty of stimulus we supposedly face. They are theoretical entities designed to solve a problem which, frankly, might not exist. There has been a lot of theoretical work on what it means to be a representation, but most of this has occurred in philosophy and most modern experimental cognitive psychologists don't ever engage with the issue. Sabrina has talked about this a bit already here and here and here and it's likely to keep cropping up.

Chemero's third chapter lays out some basics about what a theory of representation has to look like. If you're going to be anti-representational you need something to be against that isn't a straw man. This is unfortunately difficult because the concept of representation is a moving target: if I had a dollar for every conversation I've had that included the phrase 'Oh, that's not what I mean by representation' (even when it really is), I wouldn't need to submit the grant I should be working on just now. This chapter is therefore going to look for a 'minimal case' of representation that doesn't bug too many people but that isn't too easy a target.

Strap in, this chapter's busy.

What else could it be? The case of the centrifugal govenor.

Previously, I’ve dismissed the idea of mental representation because 1) no one knows what a representation is and 2) the arguments for representation tend to be pretty weak. Now, I’d like to spend a bit of time discussing a possible alternative – a dynamical systems approach to cognition. To frame this discussion, I’m going to summarise a very handy philosophy paper by Van Gelder (1995) in which he distinguishes between a computational and dynamical solution to a particular problem (see also Andrew's post on the polar planimeter). Van Gelder has clearly picked a side - that cognition emerges from dynamical systems and that cognitive processes are evolutions in the state-space within these systems. One of the main arguments for computation is that it’s difficult to imagine what else could be going on (see footnotes p. 346 for references for this argument). So, Van Gelder wrote this paper,  not to decisively rule out computation, but to provide an answer to the question “what else could [cognition] be?”

In which I finish talking about discrete computational representations

In a previous post, I summarised Dietrich & Markman’s definition of representations and ideas about how representations get their content. While there are many flavours of representation, D&M subscribe to the discrete computational (DC) variety. To summarise the previous post: According to D&M, representations are internal mediating states that govern behaviour. Representations have relations to both the external and internal (i.e., other representations) environment. They acquire content in two ways. The first way is through correspondence, where some internal state connects to some external state. The second is through functional relations with other representations. Representations are transformed via computations.

The main purpose of this post is to summarise D&M's main arguments in favour of discrete representations so that I can refer to these in other posts. I make several comments about the quality of these arguments, but this is in no way meant to be a systematic response to their paper.

Internal representation or behavioural dynamics?

More on Gibson later, but I wanted to get to this today. Yesterday I saw a talk about eye tracking, and how people control smooth pursuit movements (the tracking movements your eyes can make when you’re following something continuously). Tracking performance is a quandary for cognitive folks, because we are often very good at it. For instance, if you ask people to track a moving stimulus and record their eye movements, they will successfully foveate the target with almost no lag or erratic need to play catch up (foveating means using the fovea, the densely packed high resolution region on the retina we rely on for precise visual perception). The lack of ‘catch-up’ is the interesting bit, and cognitive psychology thinks that it is evidence of prediction by the system. Prediction requires a predictor, which for cognitive psychology is always a representation.

The main thing I learned from this talk is this: I am a rampant ego-maniac who is convinced I am right and other people are wrong, but at least I am capable of entertaining the idea that there is another way to conceive of the task. This speaker (and at least one other person in the room) was completely unaware that their perspective entailed assumptions about the underlying mechanism and simply couldn’t conceive of another way to describe the task: for them, prediction was clearly required and therefore internal representation was clearly required.

Whose representation?

One of my pet peeves is when different groups of psychologists use one term to refer to something for which they have multiple, sometimes contradictory, definitions. When I started studying similarity, I wasted a lot of time trying to clarify what everyone meant by relations. See, for one group of people, relations occurred within a stimulus (a cat’s legs are under its body, its whiskers are on its face). For another group of people, relations occurred between two stimuli (you use a hammer to hit a nail). These are very different types of relations and they affect similarity in very different ways. But, by using that one word, relation, the literature was all muddied about how relations influenced similarity. Clarifying the terms (we now speak of structural relations vs. thematic relations) helps clarify how we think about the subject.

The word representation is used in a comparably muddied fashion. Depending on who you’re talking to, representation might refer to something symbolic, perceptual, discrete, or continuous; and these symbolic/perceptual/discrete/continuous things might be transformed or acted on via ordinary computations or differential equations.

To get to the bottom of this, I want to clarify the different ways in which representation is commonly used. Then, I want to figure out how to introduce some precision in talking about representations. This will make it much easier to discuss the problem of representation and to consider the alternatives.

Today’s installment: Discrete computational representations (based on Dietrich & Markman, 2003).

Representations are internal mediating states. Anything that changes / transforms / acts on input to a system in a way that changes / transforms output (i.e., actions) is a representation.
The authors provide four conditions for this definition.

1) There needs to be at least one system, which has internal states governing its behaviour.
2) There needs to be an environment, although this doesn’t have to be the external environment. It could just be an adjacent system.
3) Some types of relations have to exist between the system’s internal states and the environment.
4) Processes must act on the internal states to satisfy goals or solve problems. Dietrich and Markman believe that these processes are computational.

On top of these conditions, the authors argue that semantic content needs to be explicit. In other words, the authors contend that psychological-level descriptions of internal states are real and that this level is more relevant that the physical-level description. Representations and processes are more important than chemicals and neurons.

How representations get their content:

1) The relations between internal states and the environment connect particular internal states with particular external states (i.e., correspondence).
2) Representations acquire some content by virtue of the types of interactions they have with other representations (i.e., functional role).

The authors suggest that 1 contributes primarily to the content of low level DC representations like a vibrating eardrum responding to sound, while 2 contributes to higher level DC representations like “hope”, “democracy” or other abstract concepts. It’s necessary for every DC representation to have at least some content from correspondence to external states.

Now, representations could be either discrete or continuous, but Dietrich and Markman argue that they must be discrete. These terms map on perfectly to the mathematical sense of continuity/discreteness. So, discrete representations are uniquely identifiable. E.g., I have a unique cat representation that is different from all of my other representations. And, discrete representations have gaps between them. My cat representation doesn’t seamlessly transition into my tiger representation (although there may be overlap).

To sum up, this notion of representation is that they are internal mediating states that are discrete and computational. Each representation is uniquely identifiable (discrete) and the processes that act on representations are ordinary computations. From now on, when I’m talking about this type of representation, I will refer to DC (discrete computational) representations.

Dietrich, E. & Markman, A. B. (2003). Discrete thoughts: Why cognition must use discrete representations. Mind and Language, 18, 95-119.

The only non-representational cognitive psychologist in the village

Hi. I’m a cognitive psychologist, but I’m not that kind of cognitive psychologist. Specifically, I don’t believe in representations, and I reject the computational model of cognition. Yes, this makes me very unpopular. I this post I want to quickly review the dominant cognitive approach and then briefly raise several potential problems with this framework. I will go through these issues in detail in later posts, but I want to go ahead and present the big picture here.

Cognitive scientists tend to view cognition as computation. In this model, representations are data structures and cognitive processes are algorithms acting on these data structures. Input => transformation of input via manipulation of discrete symbols (representations) => output. The clear analogy is to information processing in a computer. Another way to think about representations is as internal mediating states (cf. Dietrich & Markman, 2003). For instance, when I see a cup, the stuff happening in the visual system will probably be a better match to my “cup” representation than to my “glass” representation. So, I correctly identify the object as a cup. In other words, my ability to identify an object depends on consulting a discrete, internal representation of that object.

There are a number of unresolved issues with this representational stance: First, there is no theory of what representations actually are or of what information they contain. Second, many cognitive phenomena seem to defy a computational explanation. For instance, attempts to use a computational framework to model cognitive behaviours have often failed to produce anything as flexible or interesting as what we humans get up to. Third, alternative stances (e.g., that there are no discrete representations or that they are not processed algorithmically) have not been thoroughly explored. Cognitive psychologists usually take representations for granted; their existence is assumed, rarely defined or tested. This just isn’t good science. I’m just raising these points here; in future posts I’ll lay out the evidence.

My goal is to spend some time discussing these issues and to think about alternatives to representation. As a cognitive psychologist, I could get away with not understanding or caring about perception. Honestly, it just doesn’t come up much. When it does come up (e.g., Barsalou) it’s in terms of the “sensation based theory of perception” (see previous post), which we know is outdated. In the long run, I want to discuss how it might be possible to ground the study of cognition in Gibsonian perception/action. This is risky since, at the moment, I have no idea what such a framework would look like. But, cognitive psychology needs to evolve, and this is currently my best bet on how that might happen.