Friday, 21 August 2015

From Specification to Convention (A Purple Peril)

I previously laid out how specification works and why it's important to the ecological approach. Read that first, because I build on it a lot here. I also laid out the corollary of specification, that it allows that information to come to be something an organism can actually use to coordinate and control functional behaviour. Here, I think out loud about how convention might be able to do similar work, because of Sabrina's work (here, and published now here; read that paper for the extended detail on this) expanding ecological information to handle tasks such as language where specification is not always an option.

This is in part an attempt for me to get my head around some implications of Sabrina's analysis. My plan here is to develop an analogy to specification. This analogy will detail the work specification does to make information informative, then try to lay out how conventions fills this role. The goal is to see if conventions can support behaviour without needing representational help. The answer will be yes, because all the differences between law based and convention-based information are 'behind the curtain', only visible from the third-person perspective. From the first person perspective of the organism, all it gets is structures in energy arrays it can try to use to organise behaviours. Conventions place no special learning burden on the organism (Golonka, 2015) and this means that convention can support behaviour with any representational enrichment the same way specification can. (The hidden differences do have consequences, however, so I will map that out a bit.)

There are many things I have not attempted to explain and as usual this reflects my current thinking, not necessarily my final thinking. I look forward to hearing what questions this leaves unanswered for the reader as a way to move this discussion forwards.

When information is not used to perceive the dynamic that created it
The specification set-up works great for the continuous control of actions. You want to organise your behaviour with respect to the dynamics that created the information, because those dynamics are the things that define the task and therefore also define what actions are a suitable, successful complement to that task. Not all patterns in ambient energy arrays are used as the perceptual stand-ins for the local dynamics that created them, however, so the specification analysis, while very important, is incomplete.

Let's take a specific example. In the case of the linguistic behaviour speech, a person moves their articulatory systems in such a way as to create certain kinematic patterns in the acoustic array (and, as a side effect, patterns in the optic array as well, which can support lip reading and also facilitates auditory perception of speech). The relevant dynamics are those of the articulatory systems. The kinematic patterns are indeed specific to those dynamics. But, critically, successful linguistic interactions do not happen if you organise your behaviour with respect to those articulatory dynamics. If you try to behave as if the information "means" or "is about" the articulation dynamics that created it, you will fail to produce functional linguistic behaviour. So interacting with speech as a language act cannot be supported by the Kinematic Specification of Dynamics (KSD).

This is usually where people abandon ship and say 'well the ecological approach is good for perception-action but it can't handle language, I'm out'. We don't frighten this easily, so let's work this through.

To succeed in producing a functional behaviour, you need to use this kinematic pattern as if it were caused by something other than the dynamics of the articulatory system. You need to use it as if it were caused by the structure of the relevant language. This structure is not created by the laws of physics, but it's one that we as a language community have agreed on. That is, it is conventional. How can a nebulous thing like a convention shape what kind of information that structure becomes?

A First Thought, Quickly Rejected

There is no specification between words and the things to which they refer; the word "dog" has no compulsory relationship to the animal in question, as evidenced by the fact that no other language refers to them that way. However, a more ecological framing of that question might be, can there be a specification relation between a language dynamic and the pattern in the acoustic array? That is, is there one pattern per linguistic structural feature? To answer this, we would need to analyse the speech act into the correct structural features, the way we had to analyse tasks at the level of dynamics for specification to work in the continuous control of action. If there can be such a specification relation, then the organism can simply use the pattern to stand in for the language structure because a 1:1 mapping means this works out well (as per specification). 

But this won't work. Even if we constrain the scope of the relationship to a single language, it's not clear what the relevant functional unit would be for specification to work. It certainly isn't phonemes, or words. Language just doesn't seem to exhibit the kind of compulsory relationship between itself and the information created by using it that makes specification an option in perception-action. I include it here just because laying it out in order to rule it out seemed worthwhile.

Figure 1. Wittgenstein knew the deal
Given the answer was 'no' to any kind of specification relationship, what is the relationship between the pattern and the language structure? We need to identify this relationship, because it will define what is required to support linguistic behaviour. Chemero (2009) and Golonka (2015) have used the term convention as a contrast to the law based relationship underpinning specification (Turvey, Shaw, Reed and Mace, 1981). A conventional relationship is one that is sustained by agreement: if enough people act as if one thing (e.g. some language information) means another thing (e.g. some language act) then for all intents and purposes, that's what it means, and the fact that there is nothing compulsory about the form of the relationship is neither here nor there (see the linked cartoon). The word "dog" can therefore refer to the animal in question to English speakers even though there's no particular connection between the word and the thing. 

Chemero formalises how conventions do the work required to support structures being informative with situation semantics (see Chemero, chapter 6). My big concern then was that situation semantics only works if you have access to one part of the relation (either the dog or the word "dog", for example) and access to the constraint (the fact that English uses "dog" to refer to dog). Where does the constraint live, though? Chemero had no clear answer and I was left with the concern that the constraint would have to be some kind of supplementary knowledge, hardly the sort of thing I want in a cognitive system. I said at the time "well maybe conventions can be shown to do the same sort of work behind the scenes as laws" although at the time I was not confident. Well, that analysis (among a few other important things) has now happened in Golonka (2015; Sabrina will blog the paper in detail later) so it's time to think about how this plays out.

Here, I just want use the work I did in the specification Purple Peril to lay out an analogy to convention that makes it clear how conventions can work just as ecologically as laws in creating information.

How conventions can promote structures to information
Let's draw out the analogy between specification and convention with respect to how it allows a structure in an energy array to serve as information.

1. Specification is a fact of the (ecological level) laws of physics that govern how dynamics are projected into energy arrays. The ecological level is simply the world of medium sized things that don't travel particularly fast - you know, the world outside your window. 
  • Organisms learn to use information in place of dynamics and get away with it because the laws have provided a context (a 1:1 mapping) in which that substitution relationship between them is there and so operating as if it's there works out. 
  • Organisms do not need to 'know' about these laws; they don't need to carry these laws around with them to guide inferences based on those laws. We have no choice but to act using only information (the bottleneck problem) and it happens to work because of the laws. Evolution was therefore able to build perceptual systems that use information in place of dynamics because it happens to work out, and no organism needs to know why it works in order for those perceptual systems to do so. 
  • Organisms demonstrate that they know what the information is standing in for by using that information to coordinate and control appropriate behaviours. For example, I show that I "know" tau specifies time-to-contact if I use it to control interception and avoidance behaviours that require time-to-contact information to succeed. (Note: yes I know that tau is not very good information for time-to-contact, but it's still a useful example). 
2. Conventional structure does not work this way; its form is not a compulsory function of the local dynamics the way law based structure is. The law based dynamics of a fly ball can only produce one set of information variables and 1:1 mappings are an option, but you can talk about fly balls using multiple different conventional systems, e.g. different languages and the 1:1 relationship goes away. However, some critical parallels remain:
  • Organisms can use language information to stand in for what the language is about and get away with it because, in their linguistic environment, enough people act as if the relationship exists that it works well enough. The agreements can break down or change (e.g. English speakers recently agreed that 'literally' now also means 'figuratively') but this generally all holds together long enough to work
  • Organisms still do not need to store the rules anywhere. As with perceptual information, your job is to organise your behaviour with respect to the information and this only works if the behaviour you try to organise can be supported by the information in question. The organism doesn't need to know why this works; it only needs to learn that it works, and this suggests we don't have to carry language (or any conventional system) around with us any more than we carry the laws of physics. In both cases, we learn to behave in the presence of information. Studying syntax, semantics etc is to study a language's specific implementation of the convention relation, but it's not studying the things the organism needs to carry around with it in order for it to implement the relation. Situation semantics isn't the deal.
  • Organisms show that they know what the linguistic information is standing in for by using that information to select appropriate behaviours. For example, I show that I understand the sentence "Pick up the red cup" by using the information in that utterance to engage in action selection that can be supported by that information (e.g. by picking up the red instead of the blue cup). 
These claims set language up as the same kind of thing as perception-action - learned behaviour exhibited in the presence of information. What kinds of behaviour the information can support varies. Specifying information can support the online control of action because it is about the dynamics that action need to be organised with respect to. Conventional information can support the selection of action because it can distinguish between otherwise equivalent options but it cannot support online control because it isn't about the relevant dynamics. This has consequences but these seem to actually occur; we don't control actions with linguistic information and language behaviour really is much less stable than prehension, etc. So the conventional relationship is not as good as the specification relationship but it is, in principle, good enough for some things and language is one of those things. 

This analysis is just a useful first step to radicalising language. People often say to me "well that idea works for perception-action stuff because of all the information that's around, etc". People are generally ok with this idea. The analysis I've laid out just clarifies that it applies to language too. This is not the complete story; but it creates a foot in the door using something I think people are apparently already happy with, and that's all I'm looking for right now.

Further reading
A Gibsonian analysis of linguistic information
A Taxonomy of Information
Chemero (2009) Chapter 6: Information and Direct Perception: Tony argues that information in general is based in convention and that the right formalism for describing that is the situation semantics of Barwise & Perry.


Golonka, S. (2015). Laws and conventions in language related-behaviours. Ecological Psychology, 27(3), 236-250. Download


  1. I like that you (both) make a distinction between the third-person description and the first-person control of the behaviour, and that you're clear that the former is irrelevant from the perspective of the latter.

    You claim that there are consequences of the difference between specifying and conventional information: the latter is a more fallible and uncertain guide to behaviour. I'd like to press on that a bit more. I'm not convinced you've shown that the fallibility is a consequence of the way the information itself is structured.

    There does seems to be a bit of slippage here between (1) talking about conventions as formal links (constraints) between bits of sound structure and classes of objects and events, as in situation semantics, and (2) talking about conventions as implicit agreements between different speakers as to what those links are. The fact that speaking is fallible may just be a consequence of differences between speakers (2), i.e. it's a consequence of the way the world is structured; this doesn't necessarily imply that the information involved in speech is itself structured in a different way from specifying information (1).

    What's not clear, in the specification-convention analogy, is what's being proposed as an alternative to the symmetry principle. Ok, you don't want conventions to have the 1:1:1 mapping. But you're still assuming that there *is* a mapping, I think (at least, from the sound "dog" to the dog-shaped object). What is that mapping, exactly? How do you get from the word heard to the property in the world that the word is about? Can you make this more explicit, in a way that shows that this relies on something that's *in* the sound pattern, and is not just in the behaviour of the person hearing the sound? Or do you in fact deny that any such mapping exists (as I'm inclined to do)?

    1. I'm not convinced you've shown that the fallibility is a consequence of the way the information itself is structured.
      this doesn't necessarily imply that the information involved in speech is itself structured in a different way from specifying information

      Right. So the information, the structure in energy arrays, the thing my perceptual systems are in contact with, isn't 'structured differently' when conventional vs specification. First, all structure in energy arrays is specific to the dynamics that created it, because all structure in energy arrays is a projection of those dynamics governed by the laws of ecological optics. So there is no difference in kind at the level of the pattern in the energy array.

      The difference only comes up when an organism tries to use that structure as information, ie when it tries to organise a behaviour with respect to it.

      1. Using a variable as information for the local dynamics can be extremely stable when the mapping from dynamics to variable is 1:1 because that 1:1ness constrains the variability that is possible.
      2. Using a variable as information for something related to the local dynamics by a convention is less stable because the lack of 1:1ness means there's less to constrain the possible variation.

      So in all cases the information variable is simply a pattern in an energy array and is the same kind of thing across the board. The difference only shows up in the use of the variable.

    2. How do you get from the word heard to the property in the world that the word is about? Can you make this more explicit, in a way that shows that this relies on something that's *in* the sound pattern, and is not just in the behaviour of the person hearing the sound? Or do you in fact deny that any such mapping exists (as I'm inclined to do)?
      I ruled out the idea that the mapping is 'in' the sound pattern in the post. The conventional mapping lives in how the pattern is used, and the fact that when I use the pattern 'dog' to refer to a dog, I get away with it when I'm speaking to other English users.

      This whole thing is nicely embodied, actually. The convention isn't stored in me anywhere, it lives in my activity.

      A useful point: both 'specification' and 'convention' are third person analysis labels. They are important distinctions for the analysis to make, but they are *not* important distinctions for the organism. I think conflating these levels is a fundamental part of modern psychology and separating it back out is one of the most useful and important things we got from Louise Barrett's book. Funnily enough, it took me a long time to realise why she was going on about this for so much of the book, that's how under the radar and in the workings of our science this mistake is. William James knew about it, of course, but then, he knew all our mistakes :)

    3. Hi Ed,
      Sabrina here on Andrew's account (can't sign out for some reason!). Anyway, I don't think that conventionality is what makes information less reliable, I think it's the nature of the constraint governing a particular instance of conventional information. It just so happens that many constraints (experimenter goals, situational co-occurrence) are less reliable than ecological laws. But, I would advocate a task-specific analysis of the nature of the constraint before drawing any conclusions about what consequences reliability would have for the organism using the information.

  2. Andrew and Sabrina -

    Through the period of development of these ideas I've often responded with (hopefully constructive) criticisms, but I'm pleased to say that I now concur with essentially all of this post and Sabrina's referenced paper.

    I think of a simple response to sensory input as implemented in what I call a "context-dependent behavioral disposition", conceptually a neural structure that in essence matches a pattern (or member of a set of patterns) of neural activity with an action. The structure is learned from experience whether the information in the pattern is specifying or conventional. In that sense, I do see a convention as implicitly "stored" in an animal (although unbeknownst to it) notwithstanding that I agree that the syntactic and semantic features of the relevant language need not be explicitly stored. Although if one accepts Wittgenstein's "meaning is use" and assumes that in general an utterance is used in order to provoke an action, then it seems that the semantics are also implicitly stored (again, unbeknownst to the animal).