A Gibsonian analysis of linguistic information

This post is based on a talk I just gave at the Finding Common Ground Conference at the University of Connecticut. Please excuse the Power Pointy nature of some sections! You might need to Ctrl+ to see some of the images clearly.  I have made some changes from the original talk content on the basis of very useful feedback I received from other conference attendees.

What is the place of language in ecological psychology? Is language a type of direct perception? Is language comprehension direct perception? Does language have affordances?

In trying to answer these questions I discovered that some things we think of as being perceptual have a lot in common with the conventionality of language and that some language-related behaviours look a lot like perception (as typically construed). I end up suggesting that we move away from talking about 'perception' and 'language' as different types of entities and instead focus on information / behaviour relations in specific tasks.


 In general, ecological language people are happy to talk about language as having affordances. Two papers from the 2010 special issue of ecological psychology on Ecological language even have the word in their titles ("New affordances for language: distributed, dynamical, and dialogical resources" Hodges & Fowler, 2010; "Speech as the perception of affordances" Worgan & Moore, 2010). However, there is a long-standing awareness of the fact that language challenges some important aspects of ecological psychology. As Fowler (1986 p 24) says:

"As to...whether a linguistic message can be said to be perceived...from a direct-realist perspective, direct perception depends on a necessary relation between structure in information media and its distal source. But,...this does not appear to apply to the relation between sign and significance."

What Fowler meant was that because the relation between linguistic structures (e.g., spoken words) and how they structure behaviour is based on convention (not natural laws) they cannot support direct perception. It is clear that language does not fit cleanly into an ecological framework as currently defined. I argue that, rather than vaguely using the vocabulary of ecological psychology to describe language, it's crucial to identify precisely where the similarities and differences are between language and perception. I want to preserve the theoretical rigor of ecological psychology. 

The goal of the following analysis is to devise an extended framework for ecological information that accommodates language without straining or redefining original notions of affordances or direct-perception. 

Comparing language and perception

To begin, let's sum up the basic ecological approach to perception as conceived of by Gibson (1979) and reinforced by Turvey et al (1981). 

1) Affordances are dispositional properties of objects and events that provide opportunities for action to complementary organisms

2) Perception is of affordances

3) Affordances uniquely structure energy (specification)

4) Therefore, coordinating behaviour wrt information is equivalent to coordinating behaviour wrt the affordance property (direct perception)

Illustration of typical perception case:

Illustration of typical language case:

In the perception case, direct perception possible because the task (tree climbing) was about the property of the world that created the information. Behaviour was coordinated WRT the property of the world that caused the information.

In the language example, however, the task is not about the property of the world that created the information. Behaviour is organised according to conventions constrained by culture, linguistic context, etc. If we look closely at the types of behaviour that might arise as a result of receiving the verbal instruction "Turn left at the corner" something interesting emerges. The instructions can have a consequence on our actions - we can initiate a left turn at the corner instead of going straight or right. Or, we might respond, "No, I'll go straight because it's faster" or something similar. Or, we might ignore it altogether (although ignoring such a statement in this kind of conversation is very unusual in our culture). The point is, whatever consequence the information has, we definitely won't be using it to control our left turn. The turn itself will be controlled by good-old-fashioned perceptual information. "Turn left at the corner" can influence action selecting, but it can't be used to control the action. I've intentionally used "action selecting" rather than "action selection", even though "action selection" is the accepted phrase in the literature. "Selection" implies that the information is evaluated by the system, which then selects the most appropriate action. This implication should be avoided. "Selecting" implies that it is the information, itself, that selects the action by interacting with an embodied, extended cognitive system. More on this later. 

Turvey et al (1981) said:

"Our strategy, as proponents of Gibson's ecological approach, is to argue for a conception of natural law that allows meaningful relations between organism and environment to hold. Further, we constrain our use of the term 'perception' (and thus, of course, 'direct perception') to relations governed by such laws."

The view of language that I propose reserves these terms for cases where these requirements are clearly met. Does this mean that language is outside the remit of ecological psychology? No, because for Gibson, the primary thing is information. And, Gibson (1979 p 255) offers a useful suggestion for how to progress our understanding of things like language.

"Perhaps if [so-called higher mental processes] are reconsidered in relation to ecological perceiving they will begin to sort themselves out in a new and reasonable way that fits with the evidence."

We've identified a key difference between perception and (at least some) language-related behaviours. Bearing this in mind, let's now go back to the beginning - which, for Gibson is that the environment is informative - and build up from there.

Re-analysis of information

1) Objects and events structure energy

2) These structures are lawfully related to the properties in the world that cause them

3) Information is any structure in energy that has a consequence for behaviour

4) Not all behaviours are organised wrt the property of the world that caused the information

5) Nervous systems are for linking information and behaviour

Point 4 simply states the difference identified between the perception and language examples above. The first 4 statements broaden the field of information. The 5th statement claims that linking information and behaviour is precisely what we are built to do - variations in nervous system type and complexity can be understood in terms of their consequences for linking information and behaviour. 

Perception / action is a particular type of information / behaviour relation. As I intend to show, calling some types of things 'perception' and other types of things 'language' obscures  important characteristics that more clearly define information types. This means that the key objective is to accurately characterise relationships between information and behaviour. What factors influence the learning of various information / behaviour relations? What type of behaviours is a type of information capable of supporting? What types of information is a particular organism capable of detecting or coordinating its behaviour WRT?

As a starting point, I've identified the following factors as important.

Evolutionary factors

• the types of energy an organism is sensitive to
• biases for an organism to pick up certain types of information
• predetermined or strongly constrained relations (e.g., reflexes / instincts)
• sophistication of nervous system - ability to learn new information / behaviour relations, to use information of varying degrees of reliability, stability

Learning factors

• must the organism learn to detect the structure?
• must the organism learn to organise its beahviour WRT the structure?
• is the behaviour organised WRT the property causing the structure or convention?
• how stable / available / reliable is the structure?
• is the information used in action selecting or in the continuous control of action?

These factors can be used in the form of questions to ask about the types of general information / behaviour relations an organism demonstrates (evolutionary factors) and the types of information / behaviour relations that are specific to a particular task. I'll illustrate how this works using just the learning factors.

The example of a human infant learning to walk is a paradigmatic perception / action task. All typical perception / action tasks should have similar answers to the ones above.

 Notice that the language-related behaviour of postural entrainment looks just like the infant learning to walk from this perspective. Thus, the information used to control postural entrainment during conversation is the same type as the information used to control locomotion.

Not all language-related behaviours are cases of perception / action, though. As we saw earlier, responding to the instructions "Turn left at the corner" involves organising behaviour according to convention. This type of information influences action selecting, but it cannot be used in the continuous control of action.

Interestingly, producing the phrase "Turn left at the corner" looks a lot more like a perception / action task. Whether the task involves action selecting or control appears to depend on the level of analysis. At the phoneme level, it seems to be about action-selecting (this is based on a talk at the conference by Brian Gick which seemed to show that, at least, some of these units were produced with only feedforward control). I don't know a whole lot about this area, but it's worth identifying that the level of analysis matters here.

This example focuses on bees who observe the Waggle Dance and then search for food in a particular direction as a result. This is the first example where the answer to the first two questions is 'no' and it shows the work that evolution can do in constraining conventional information. Still, this conventional information is only involved in action selecting, not in online control. 

Other behaviours that do not require learning to coordinate behaviour wrt the structure include reflexes and (some) instincts. Reflexes and instincts differ in that information for reflexes is for action selecting and information for instincts is for action control. 

These questions can also be applied to conditional learning, as in this example where bees were trained to associate a particular colour with food. What's nice about this framework is that it provides a common vocabulary for looking at the whole range of animal behaviours.

Quick re-cap and implications

Perceptual information, ecologically defined, does not encompass the full range of information / behaviour relations. In particular, it excludes cases where behaviour is organised according to convention, as with some language-related behaviours, animal communication systems, and conditional learning. Some language-related behaviours (e.g., language-related entrainment behaviours) are controlled by perceptual information, as defined by Gibson & Turvey et al. And, some non-language-related animal behaviours (e.g., behaviours resulting from operant conditioning) are selected by conventional information. This means that a clear perception / language divide is misleading. Information / behaviour relations are more clearly organised according to the factors that influence the learning of those relations, as in the examples above. 

A major implication of this analysis is that many aspects of language do not "have affordances." Affordances are dispositional properties of objects or events that structure energy and provide opportunities for action to organisms with complementary effectivities. Their relevance to ecological psychology is only to establish that there are properties in the world that can support the online control of action. However, not all information / behaviour relations are about the continuous control of action - many are about action-selecting. The theoretical work that affordances need to do is not required in these cases. Nothing is gained by trying to broaden the definition of affordances to accommodate these examples.

A second implication is that many aspects of language do not qualify as direct perception. Direct perception is specifically when behaviour is organised wrt the property of the world causing the information because this entails that organising behaviour wrt the information is equivalent to organising behaviour wrt the property of interest. People have begun using direct perception as shorthand for "not mediated by representations," but it is only one way of being non-representational. Acknowledging that some aspects of language are not examples of direct perception does not necessitate a representational account of language. Nothing is gained by trying to loosen the definition of direct perception to accommodate these examples.

Finally, while this analysis draws out many differences between language-related behaviours and traditional perception / action examples, these are really just two regions on a single field of ecological information. For all learned information / behaviour relations, from walking to talking, an organism must learn to organise its behaviour wrt information in the environment. To re-iterate the point made earlier, this is what nervous systems are for.