Friday 25 May 2012

Language: A task analysis (kind of)

In the last post, I discussed the similarities and differences between language and other types of information. From the first person perspective, spoken language is just another type of auditory event. The main distinction between the word "dog" and the sound of a dog barking is that the auditory event of barking is about the thing that caused the sound - a barking dog - while the auditory event of the word "dog" is not about the thing that caused the sound - a human speaker. The word "dog" is (usually) about an animal that is related to the auditory event by convention. Thus, the sound of a barking dog conveys auditory information and the sound of the word "dog" conveys linguistic information.

In this post I want to lay out classes of tasks in which linguistic information is useful. As a starting point, I will identify situations where language appears to fill a gap, although at this point these are no more than general descriptions. In any specific task analyses that might eventually follow, the basic strategy will be to begin by asking what perceptual resources exist to carry out the task. If perceptual resources are unable to explain task performance and if linguistic resources are available, then these will be considered for their potential contribution. It might be helpful to think of linguistic and perceptual information as occupying different niches in a task space. Perceptual information helps me to walk and catch a fly ball and linguistic information helps me do the types of things described below.

1. Expanding the range of knowledge about the environment

Perceptual information is limited in time and space. From a given point of observation, I can perceive some things about the environment, but not others. I can see a tree that is in my direct line of sight, but I can't see the dog behind it. I can hear the dog bark behind the tree, but I can't hear the bark of a dog 10 miles away. For the auditory event of a barking dog to be informative I have to be close enough to the dog at the time it is barking. If I am close to the dog but it is silent or if I am too far away from a dog that is barking, then I will not learn about the dog's presence via auditory information. Similar limitations apply to all modalities of perceptual information. This means that our ability to learn about properties of the environment via perceptual information is always limited in time and space by our ability to detect the information. This constraint is due to the nature of the mediums through which perceptual information flows (e.g., air, water).

Linguistic information is not similarly bound by time and space. A language user must be within earshot of auditory linguistic information (or within sight of visual linguistic information), but that speech event can be about properties of the environment that are not perceivable from that point of observation. Thus, linguistic information extends the range of properties we can know about an environment.This is due to the nature of the medium through which linguistic information flows (language) because the relationship between information and what it means (although, see my below for me waffling over the use of the word "meaning" in cases like this) to be one of convention rather than specification and this allows the meaning of the event to be different than the physical event that causes the linguistic information (e.g., a speaker).
2. Conducting speech acts

Austin introduced this idea in his book "How to do things with words." Essentially, there are some actions that cannot exist without language. For example, it is impossible to get married in the US without the words "I now pronounce you husband and wife." The act of marriage is accomplished via these words. It is impossible to make a promise to someone without language (or some other symbolic act that has a conventional meaning of promising). 

3. Recording information

This is essentially an extension of point 1 above, but it is such a specialised case that I thought I'd give it its own category. Language allows information about the world to be preserved over very long time scales and distances through songs and writing. This function of language critically allows humans to build upon prior knowledge and is one source of stability in culture.

4. Directing behaviour within the perceivable environment

Linguistic information can be used to shape behaviour within an environment. A person who is seated in front of two equally graspable objects can reliably be made to choose one rather than the other on the basis of linguistic information (e.g., "pick up the red one"). It is interesting to note the difference between linguistic direction of behaviour and the compulsory effecting of affordances. If I perceive an affordance and if I am disposed to effect that affordance (by having assembled the right kind of task specific device), then I must effect the affordance (yes, yes, I know this is contentious. Regardless, acting on perceptual information is more obligatory than acting on linguistic information). This is not the case with linguistic information. If some one tells me "pick up the red one" I don't necessarily have to comply. However, linguistic instructions often do work because humans are social animals with complex notions of behavioural appropriateness and obligation. Understanding this class of tasks will necessitate a good understanding of the social embededness of language.

These four categories are about things that can't be accomplished very well without language or some other form of symbolic communication. To accomplish #1 and #3 you have to be able to refer to things in their absence, which means that you have to have something to hand (words, gestures, etc.) that can substitute for things that aren't there. The actions described in #2 only exist in the medium of language. #4 is a borderline category. It is possible to use gestures to direct someone to a particular object in the environment, but, you can direct people more accurately (and to do more complex things) if you also have language.

The next three categories are about things that don't necessarily need language, but that we use language for nonetheless.

5. Social relations

Humans are extremely social animals. Not only do we need each other's company, but the network of relationships we form is large and complex. Apprehending the meaning of linguistic information can cause changes in emotional states in ways that maintain, strengthen, or break social bonds.

6. Self-regulation and loopy cognition 

One of the interesting things about linguistic information is that we can create it for our own use. I can tell myself to stop wasting time on the internet, I can repeat the items on a grocery list until I get to the store, I can recite a mantra during meditation. The creation of this type of linguistic information changes our own behaviour - I get back to work, buy the right stuff at the store, or slow down my heart-rate. The parallel case in perception/action is our ability to move around, which changes the landscape of affordances in the environment.

7. ???

I don't know quite how to capture this category at present. We talk to dogs and washing machines and thin air. We talk to things that can't possibly understand what we're saying. At first I was tempted to lump this in with #6, but I think there is something distinct going on here. We don't actually expect dogs to understand what we're saying, but talking to animals seems to be about communication at some level anyway. This makes it distinct from self-regulation. Talking to / yelling at inanimate objects cannot really be an act of communication, but then why do we do it? Is it simply a way to let off steam (which is self-regulatory)? Perhaps language is just one of the mediums through which we channel energy -  humans simply natter on.

Perceptual-linguistic systems

Our use of perceptual information is context-specific. The online control of locomotion involves perceiving terrain with respect to our current capabilities. Just looking at an environment doesn't tell us about whether we can move through it; this information must be calibrated with respect to our bodies and current state. Thus, our actions (e.g., can I walk over this surface?) reflect perceptual and embodied factors.

Another way to say this (although not the way it is usually talked about) is that the meaning of an information variable is inherently multidimensional. Although from the 3rd person perspective, we can examine an information variable and see that it specifies, say, the catch-ability of a fly ball, from the 1st person, perceiving a variable in itself is meaningless. In order to learn the meaning of / how to use an information variable, we must perceive it in concert with other perceptual information that tells us the consequences of undertaking a particular action with respect to that variable. Even in a trained system, other information always impinges on the specific meaning of an information variable in a given context. For example, the state of my body (exhausted versus energised) changes whether an information variable about catch-abiltiy means catchable or not catchable (this is accomplished because the state of my body changes how I am able to move with respect to the variable). Thus, the specific meaning of an information variable is multidimensional (relies on multiple sources of information) and context-specific.

Our use of linguistic information is also context specific. In this case, the meaning of a particular speech event arises via a system comprising both linguistic and perceptual information. Linguistic information is always embedded in a broader perceptual context and the way we use linguistic information is inseparable from this context.

Consider the sentence "I am going to kill you" in the following contexts:

Said by a stranger hiding in the alley

Said by you when your partner forgets to take off work to see your daughter's play

Said by your partner who was just startled by the surprise birthday party you arranged for them
The way the sentence "I am going to kill you" influences our behaviour (or, what it means) is mostly dependent on the broader perceptual context in which it appears.

Here is an example taken from the glorious tumblr, Sh*t my students write that illustrates the same point:

Eye contact can show you how much the person means what they say. For example if a husband looks his wife in the eye and says “I love you” it will come across better then him looking at his car and saying it even if he sometimes loves his car more. 

Indeed. An utterance does not have a core meaning that is modified by context. Meaning arises from perceptual-linguistic systems that evolve over time and space.

This presents an apparent paradox. Earlier I said that a speech event and its meaning are related via convention. Now I am saying that the meaning of a speech event is also determined by perception (of the current context) and that a given utterance has no core meaning. The seeming paradox arises because it is tempting to equate the notion of the conventional meaning of a speech event with a definition of a word (e.g., the meaning of the speech event "dog" is equivalent to the definition of the word "dog"). In ecological psych, meaning is defined in terms of appropriate use. So, if I can use the word "dog" correctly, then I know what it means.

** As an aside, our cultural view of language is strongly bound up in the idea of definitions. The longest books our culture produces are lists of definitions called dictionaries. But, consider the fact that it is possible to be fluent in a language without knowing a single definition. When children learn language it is always in the course of doing things. You don't sit a baby down with a dictionary and recite the definition of "water" to her. You say the word "water" whenever it occurs to you that it is relevant in the course of daily life. It is surprisingly easy to catch people using words correctly that they can't define - "nonplussed", "enervate", and "penultimate" are fun ones to try. These are unlikely to trip up readers of this blog, but the point is that language works perfectly well without the concept of definitions. And, without the concept of definitions permeating our culture, it is interesting to reflect on whether linguists and psychologists would have felt the need to invent the idea to explain language use.**
Since speech events are not about the physical production of the speech sounds, whatever meaning such events do have will be conventional. This does not necessitate that the meaning will be the same across contexts. In fact, the further I delve into this question the more I am convinced that meaning only exists at the level of perceptual-linguistic systems. There is, of course, some stability in the way a word is used a across contexts. The traditional cognitive approach is to try to explain such stability by invoking concept representations. A successful ecological explanation for language use must also be able to explain stability, but, to me, this isn't the interesting bit. A more important line of investigation is how perceptual-linguistic systems facilitate successful communication given variability in meaning.

In fact, look, if I were to go back and re-write the last post I wouldn't talk about meaning at all for either perceptual or linguistic information. I would only talk about appropriate use - do we know how to use words to change the world in ways that suit us? Ecological psych talks about "direct access to meaning" in terms of what happens when we perceive an information variable. But, what this really means is that we are able to coordinate our action appropriately with respect to this information variable. This is the same thing we need to know about language - how do we coordinate our actions with respect to linguistic information?

A note on representations

It is worth stating explicitly that this take on language is non-representational. Representations are not needed to explain how linguistic information can guide behaviour and be informative about the world. Establishing a parallel with perception-action helps to explain why. 

In perception, we have to learn to detect information variables or event structures and then we have to learn the meaning of / how to use these variables or events. As perceivers (and not psychologists adopting a 3rd person perspective) we don't know that these variables or events are in a specification relationship with the environment. The fact that they are in this type of relationship means that we can use these variables in the continuous control of action (because they are sufficiently stable). But, as perceivers we don't know anything about specification and we don't need to. As perceivers, our job is to figure out how to use information appropriately (i.e., to figure out what the information means). We don't get access to this for free; we have to learn it over time. 

Ecological psychology doesn't pay enough attention to the acquisition of meaning and this can make it seem like detecting an information variable is always equivalent to knowing how to use it, but this is true only for a trained system. Exactly the same problem is faced by the language learner. First we learn to detect linguistic information and then we have to learn how to use this information. Although perceptual information means what it means in virtue of its relationship to the physical world and linguistic information means what it means in virtue of its conventional use, the task facing the learner is identical. Differences arise not because perceptual information and linguistic information are fundamentally different things but because they occupy different niches and probably because what constitutes appropriate use of linguistic information is much less stable than what constitutes appropriate use of perceptual information.

This parallel means that if representations aren't required for perception-action, then they aren't needed for language either. People are generally more willing to accept a non-representational account of perception-action because it jives with our intuition about how things work. When I walk down the street I don't need to recall my walking representation, I need to look where I'm going. Many people's intuition about language is that we do have to recall/activate a word's representation in order to use it. But, perhaps, like walking, language is also basically about looking where we're going. In any case, it's not good science to rely on what is intuitive and to reject whatever seems odd. There is a good argument that  perception-action can work without invoking representations and there is a solid theoretical framework for understanding how it works without representations (ecological psychology). If language is the same type of thing as perception-action, then it too will work without representations. We should not dismiss this logic because it seems weird or counter-intuitive to think of language in this way.

To summarise the main points:

1. We talk to ourselves, to things that don't understand us, to things that do understand us and we talk to us/them ABOUT language things, social things, stuff in the room, stuff out of view and stuff that doesn't exist IN ORDER TO change something about the world to suit us

2. For a given task, the first place to look to explain performance is perception. If perception cannot explain task performance and if linguistic information is present, then consider how linguistic information is being used

3. What constitutes appropriate use of linguistic information emerges from perceptual-linguistic systems rather than from linguistic information alone.

4. If perception-action is non-representational, then language use is non-representational

This post and the two previous ones summarise my attempt to explain language use within an embodied and ecological framework. There are many, many unknowns still, but now begins the fun part of actually doing some research to see whether the data support these ideas (BTW, there will be some fully funded PhD studentships on this topic advertised shortly). In terms of the basic approach, I still need to work out the kinks in how I talk about meaning. I could have spent another week or so re-writing this to make the language more precise. But my maternity leave is ending and I'll be back to work soon so getting out something is better than nothing!


  1. An interesting account, but I'm not convinced by this bit which seems to be the crux -

    When I walk down the street I don't need to recall my walking representation, I need to look where I'm going. Many people's intuition about language is that we do have to recall/activate a word's representation in order to use it. But, perhaps, like walking, language is also basically about looking where we're going....

    I certainly don't think that we need to recall the meaning of a word in order to use it. If I (for example) bang my head on something and shout "Ow, F**K!", that is pretty much a reflex response and has nothing to do with the 'meaning' of that word - a word which I'm not sure has any single meaning at all.


    While much language-use may go on without activating representations, the fact remains that many words do seem to activate representations, even when they are not being "used" at all.

    For example if I came across a piece of paper with the single word "DOG" written on it, an 'image' of a dog would probably 'come to mind'. I would have no idea what the purpose of this piece of paper was - I would have no use for it - but it would cause me to think about dogs.

    Not all words do this. If the paper said "THE" then no such 'representations' would come to mind. "The" is a word which, one might say, has many uses but no meaning as such. As such I think your approach would have no trouble with "the" and similar words, but what about nouns? Surely they're different?

    1. **I haven't thought about this for long** but...
      These look to be different cases. The first is "language in use" - so the 'f**cking' case, or any other case where I produce language to afford some response in the world (inc. myself).

      But the second - a word conjuring up a representation - is different if it's from an external source, in the sense that we assume the "language in use" is "language to produce some effect in us".

      In the first, my use of language is responding to a thing in the world, in the second the representation is on the page - the word - my response to it is attempting to make sense of the context in which it appears using the set of perceptual information at hand, so seeing "dog" on a zoo cage (strange zoo) would lead to one action set, while seeing it on a quarantine cage might another, next to a picture of a dog and "chien" another, etc.

      ...and then the stuff about DSA v. representations which Sabrina talks about below (btw, some of Andy Clark's stuff on moral cognition touches on that I think, I wrote an essay on DSA, Clark and a sociocultural guy called Tappan, linked to a special issue of 'journal of moral education')

  2. More on that: proper nouns seem to be the hardest challenge for your system. "Nick Clegg" for example clearly refers to a particular person.

    To the extent that hearing the term "Nick Clegg" conjures up a mental image in my mind (a face, a voice, etc.), then, surely I must have an internal representation of Nick Clegg? And supposing I had never heard of Nick Clegg, and then one day I saw him on TV and learned who he was - surely I must be gaining a representation of Nick Clegg?

  3. Thanks for following up on this idea that you raised in the last post. It is very useful to practice responding to this, since it will be the most consistent criticism I get from the cognitive camp.

    First, I think it is important to clarify some terms. It is essential to be clear about what you mean by representations and to specify what work they expect them to do in cognition. If you don't know the answer to these questions, you wouldn't be alone, but it does question the merit of feeling like such vaguely understood entities must belong in a theory of language.

    A typical perspective is that representations are internal mediating states. Some representations carry content and some perform functions. They are discrete entities. The purpose of representations is to transform input from the environment in some way to produce an appropriate response to output. This is your typical information processing account. See these for a summary:

    Onto your point: When you say that "the" doesn't require a representation but that "dog" does, what do you mean by this? What problem in understanding "dog" is being solved by a discrete internal mediating state? I agree that we have a subjective experience of some content coming to mind when we thing of "dog" or "Nich Clegg". But, this doesn't mean that this content exists in the form of a representation.

    My account allows leaning to occur (obviously the system must be able to change over time) and it allows internal mediating systems (rather than discrete states)since stuff clearly happens in the brain - neither of which is at odds with having a subjective experience of things coming to mind when I think of the word "dog" or when I lean about a guy named "Nick Clegg". So far, I haven't gone into an ecological perspective on this experience but this doesn't mean that the account can't handle it (it's just not the focus at the moment).

    Now, this is when, in other posts, people really want to say "so aren't these just representations?" No, they're not. Systems are not discrete entities that are transformed via cognitive operations. They continuously evolve over time and space and do not "represent", in the since of "stand in for", the environment. See here for a detailed analysis of how a dynamical system is different than a representation:

    Or, have a read of this:

    Van Gelder, T. (1995). What might cognition be, if not computation. The Journal of Philosophy, 92 (7), 345-381.

    It is not a legitimate move to simply redefine "representation" so that some form of the concept survives in whatever theory (or to invoke them because it feels as if something like them must be there). They have to do some work in cognition or they are not required as theoretical entities. I agree that abandoning core word meanings defies intuition, but there needs to be a more substantial theoretical reason for that to invoke representations.

    And in terms of the diagnosticity of having a subjective experience of mental images in supporting an argument for representation, see here:

    Have a read through the comments - there are loads of people who don't have this subjective experience, but yet are able to think just fine.

    1. On the topic of what the brain is up to, if not representing: I've posted a few thoughts on this in these posts. As Sabrina says, clearly we have internal systems modifying the flow of information (perceptual and,for humans, linguistic) and these internal systems matter. But they are not representations and it's not useful to cling to the word for the reasons laid out in Sabrina's other posts on this.

      This is useful, though; it's clearly time for us to start integrating some of these pieces again because this is going to be a common critique.

    2. Another approach to your question is this:

      I've laid out (what I think is) a logical parallel between perception-action and language use and conclude that, because of this parallel, if representations aren't needed for perception-action, then they aren't needed for language either.

      If you disagree with this, it would be helpful if you could identify either an error in my logic or an error in my premises. It's not that I'm so confident at this point that I don't think you'll find anything, but taking the discussion forward in this way will, I think, be more fruitful in terms of honing this theory.

    3. neuro -

      A major problem I see with trying to interpret mental imagery as representations is that doing so is trying to explain the not-well-understood in terms of the not-at-all understood. As far as I know, there is currently no even remotely convincing hypothesis on how mental imagery works, nor even what it's for (in case you didn't follow the last link in Sabrina's reply directly below your comment, be sure and check it out here). That and sensory substitution seem to raise the question of its utility. And despite trying for many months to think of something that actually requires it, I've come up with zip. The same neural activity from which images must be derived seems in principle adequate by itself for any function I've considered.

    4. The perception-meaning-language pathway/cluster seems to need a representation to form a connection in the brain, whether this representation is "visualised/mentalised" or not, I would think. Have you studied the work by Kosslyn & others on the two different neural pathways that allow mental representation "in the mind's eye"? The two tracts or pathways are said to represent different aspects of the image- it's "whatness" & "whereness"- 2 aspects of some sort of representation. Apparently people with good imaging ability perform differently on 3-dimensional imaging tasks of figures rotated or oriented differently, as in some intelligence tests. Also, the natural "salience" of environmental cues such as size, shape, distance, brightness and relative orientations & illuminations must form the basis of any early perceptual learning. I'm trying to illustrate that some sort of representation needs to occur in ordinary people for them to manipulate ideas about the world they find themselves in as infants. If they can't work out what they're supposed to pay attention to & form a representation of it, then they won't behave or communicate the same way as everyone else. Anyway, I won't drag you into my territory any further, but I'm trying to suggest that an internal representation of some type, whether explicitly sensory or not, is needed in a practical way so that the represented "thing" can be mentally manipulated with other "things". Sometimes with a learnt language, sometimes as a hard to define "thought object".

  4. Hi guys. I'm @inkblurt on the twitters, and have been pinging you on occasion.
    I'm in the course of writing a book on how information creates/changes context. (I'm not an academic, just a practitioner who has been working on this issue for a long time; but I'll stop there re the book)
    Anyway, I'm in the midst of writing a chapter on language now, and these posts have been immensely helpful for me to clarify some things in my head, namely the relationship of information to language.
    As someone who wasn't very deeply immersed in the conventional view (i mean, culturally I was, but not ideologically, so to speak) it's fascinating to comments from people who are. I remarked to a friend that it feels like the stuff followers of Ptolemy's way of seeing the universe were trying to explain "anomalies" in the paths of stars / planets before the Copernican point of view finally took hold, and explained it all much more directly and elegantly. But to a Ptolemy-pov critic, it must've taken a great deal of work to finally realize just how deep your assumptions go in skewing your view of the Copernican model.
    Anyway, about this language thing.
    The direction I've been running (based on the work I've read in Barrett, Clark, etc and in your blog) is that language is a sort of environment, in the Gibson sense of environment. But it's an emergent environment we've created collectively as a species. It's sort of like a simulacrum that's generated not by external third party, but through tacit collective emergence from social interaction. I think of it as a sort of dimension we live in simultaneously with the physical dimension (or, i.e. the only one we would be living in if we didn't have language). The distinction is actually "academic" I think -- to the organism, there's not a big difference. We probably distinguish them mainly after the fact (as our self-reflective reification-prone new-brain stuff is wont to do).
    In the case of "dog" vs "promise" -- it's arguable that, as a species that depends so much on other people for our survival, from infancy onward, "promise" is something we encounter on a regular basis even more intimately & profoundly than we encounter a physical animal-object "dog." A young child depends on the promises of the surrounding adults much more than on a dog.
    We live in and perceive-act with language, we walk around in it, as a sort of information that's just as vital as the environmental information Gibson describes. I don't just 'get' language -- I have to move around in it, poke at it and see what it does, take into account the surrounding context, etc. I don't have to cognitively process conversations (in fact, we suck at that -- I had to learn how to observe my communication like that in therapy!) I just interact and react. Logic is an afterthought.

    Also - I've been reading Terrence Deacon and I realize (at least his Symbolic Species book) is still getting tangled up in the representational issue, but I wonder if his evolutionary approach has any relevance to your arguments?

    Ok sorry for the very long comment (or, perhaps, missive) but I'm very engaged in this material -- it's sparking all kinds of stuff on my end. Thanks again.

    1. Hi Andrew H,

      Thanks for your comment. The book sounds quite interesting. I very much like the idea of language as an environment we live in. This was the idea I was hoping to convey by describing language as a medium. I also like your emphasis on language as something we need to explore, "poke it and see what it does." I think one of the reasons humans are able to learn language (which, admittedly, is hard work)despite its unhinging from specification is that we can make lots and lots of mistakes usually without the risk of bodily harm. I've noticed this on the blog many times. I will write something that I assume is crystal clear, only to have every reader interpret it differently than I intended.

      I'm reading Daniel Everett's book "Language: the cultural tool" and it seems to get at some of these ideas from an evolutionary angle and that is definitely something I've been thinking about. I don't know Deacon's stuff, but I'll check it out. My current thinking is to try to come up with a solid theory of language use as it is now before working too hard at the evolutionary angle. The reason for this is that evolutionary psych is extremely difficult to do properly and without a really well-defined evolutionary research programme we run the risk of coming up with Just So stories to explain our preconceptions.

      Anyhow, rather than talking about language as a medium or environment, Everett construes language as a cultural artefact. Aspects of his theory are still quite representational (he talks about brains needing to decode linguistic information), but he is thinking about what language is good for and how this function might shaped its invention and development.

    2. @sabrina Thanks for the reply! I'll look at Everett's stuff as well.

  5. Neuro,
    Ignoring the language issue proper, could we get some more scare quotes in your claim. For example:

    To the extent that hearing the term "Nick Clegg" conjures up a "mental" "image" "in" my "mind", then, surely I must "have" an "internal" "representation" of Nick Clegg?

    I am not doubting that there is a very real phenomenon you are asking about, just pointing out how loaded your description of the phenomenon is.

    IMHO, the best bet for a realist view on this is to assert that one function of language is to make you respond to things that you do not directly sense (i.e., to reconfigure you into the type of devise that is responding to those things). This is compatible with Sabrina's analysis above - as well as certain analyses of communication in animal behavior. As part of that, you may come to respond to a person whom you have seen before, though they are not here at the moment. You are asking (I assert) about part of that extended response, which can occur with or without language as a trigger. Language does seem to be very good at arranging such things, but it is not unique in doing so.

    Andrew and Sabrina don't seem sold on that. I would bet they are even less sold on the possible extension of that idea to claim that your so-called "mental image" is perfectly well explained as an extended response to the visible aspects of the person.
    P.S. Andrew, I just realized that I am comfortable with the word "reconfigure"! This makes it even more mysterious why I am so uncomfortable with "reassemble".

  6. Oh, and proper nouns are a big challenge for any direct perception system - language or otherwise. There are many ways to try to deal with the problem. Some (but not all) require completely undercutting our notion of proper nouns.

    The reason it is not unique to language is that the same trouble applies to any other situation in which it seems important that an object be a specific object, e.g., my drinking cup or Andrew's office door.

    1. Is this related to Chemero's worry about direct perception of individuals? That struck me as a complete non-problem.

    2. Two two are related (possibly identical).

      I think it is a problem, but I also think it is a problem that it is very reasonable to put off until later. I recall not ever feeling like we resolved that prior discussion. Certainly there is nothing fundamental or basic about the "perception of individuals". So either the problem will get redefined in the new system, it be handled as a special case, or it will be considered a phenomenon that is governed by different (but compatible) rules.

      As I indicated before, the problem is that all cups at the party afford my drinking out of them, but for some reason I only drink out of my cup. There is nothing in the optic information, or any law-like elements of the me-cups-party system more generally that should lead me to selectively drink from the cup that is "mine".

      We should get Tony and Neuro in on a dialog-style article for Ecological Psychology about this. The material in the prior post is almost enough to justify that already.

    3. I'm jumping in here without knowing the context of the discussion you had with Andrew, so my apologies if you've already been through this or if this misses the point of the argument.

      It seems to me that the confusion about individual instances arises because of the focus on affordances while ignoring other perceptual properties. Part of the reason that all cups are cups is that they share affordances. But, affordances are only a subset of perceptual properties - the ones that are action-relevant. These are the properties that ecological psych is most interested in because they can be used in the continuous control of action. However, they are not the only things that are perceivable. The colour of a cup is not related to its function, but I can perceive it. I know which cup is mine at a party as long as I can discriminate between it and other members of the category "cups". It might be a different colour than the others, or I set it in a particular location. If there is no perceivable difference between my cup and other people's cups, then I likely won't end up drinking from the right cup.

      Arguing that it is problematic for me to selectively drink from a cup that is perceptually discriminable from other ones in the vicinity seems to ignore the role of learning. No, there is no particular reason that I received a particular cup, no lawful relationship there. But once I have one, why is it problematic that I learn to identify it on the basis of perceptual properties?

      Feel free to point me to the post where you all were discussing this before - I'm genuinely interested in this debate.

      Of course, there is no lawful relation between that cup and the fact that I ended up with it, but I think the expectation that there should be one is a misreading of the role of lawful relationships in perception. The lawful relationship is between structure in the energy array and the physical world. The relationship between all the perceptual details of my cup (size, colour, pattern) and the world will be lawful.

    4. FWIW, I second Sabrina's position. This an example of the point in my comment below about the non-optional character of affordances when total context is taken into account. We are taught from early on not to eat or drink using other people's dishes, utensils, etc. A stickler for sanitation doesn't have the option of choosing between say, the closest cup and the more distant cup that's "mine", whereas someone who doesn't care and is terribly lazy does. In each case, only one cup offers - or after some intermediate move can be made to offer - an affordance.

    5. I think, if we nail down some technicalities, that I agree with Sabrina.... I'm interested to see what Andrew thinks ;- )

      The problem arises out of the particular brand of eco psych promoted by Turvey, Shaw, and Mace (TSM), in which it is argued that all perception is perception of affordances AND all perception is governed by natural laws. If you are a fan of that system, one place it causes trouble is in trying to explain my reaction to individuals. It might be possible to overcome the trouble, but it gets ugly.

      By my read, Gibson did not advocate so strong a position, and so he does not have any particular problem here.

      Andrew had the link above, but here it is again:

  7. "In ecological psych, meaning is defined in terms of appropriate use. So, if I can use the word "dog" correctly, then I know what it means."

    Thank you for making such an explicit statement on this. This clears up your position from your last post. Although, you said something toward the end of this post that brought up some more questions for me:

    " I would only talk about appropriate use - do we know how to use words to change the world in ways that suit us?"

    What is encapsulated by "use"? Everything that you know about a word would include phonologial, morphological, syntactic, and semantic information. Let's take the word "butter" for example. If you speak English, you know that you can pronounce the /t/ as a real coronal plosive [t] or as a flap [ɾ] (a kind of quick d-like sound), which is only due to the position of /t/ between a stressed and unstressed vowel (you know not to ever flap the /t/ in "tower" or "attack"). You know this can act as a noun or a verb, which also entails that you know how and when to use articles with it, how to pluralize it, how to make it agree with a subject, how to put it in the passive, how to negate it, etc.

    I guess what I'm asking is quite a big question: is there any room for "mental grammar" in your model? How do you do grammar without symbols?

    "As an aside, our cultural view of language is strongly bound up in the idea of definitions...And, without the concept of definitions permeating our culture, it is interesting to reflect on whether linguists and psychologists would have felt the need to invent the idea to explain language use."

    You've emphasized dictionary definitions a couple of times now, both here and to make some points about dog vs. promise. They don't really play the role in linguistics you seem to think they do, and I certainly contest your claim that we invented them for our own purposes. Meaning is inferred by empirical study of native speaker behaviour and intuitions. Words mean what people use them to mean, regardless of what a dictionary says. Formal work in linguistics is usually concerned with denotations, truth conditions, or conditions of use, not definitions. Field work in linguistics (i.e. language documentation) obviously happens without recourse to dictionary definitions, because we may be studying morphemes whose meanings are still unknown. Besides, it's common for linguists, semanticists even, to work on languages with no writing system, let alone a dictionary. Language use can be and is studied without invoking dictionary definitions.

    1. "I guess what I'm asking is quite a big question: is there any room for "mental grammar" in your model? How do you do grammar without symbols?"

      "Everything that you know about a word would include phonologial, morphological, syntactic, and semantic information"

      In answering this, I would, in turn, ask you "What do you mean 'know'?" If you simply mean that a person uses language in a way that can be formalised by linguists as having these properties, then it fits very well with my account. If you mean that a language user has representations of these levels of linguistic knowledge, then we are in disagreement. The example I always draw on is the Watts governor. One way to describe this machine in action is that it "knows" about the speed of the flywheel and adjusts itself accordingly. In other words, the system appears to operate according to a set of rules. This is the typical way to talk about language and it seems to require representations (somewhere to keep the knowledge of the rules). But, the Watts governor is a non-computational device. The "knowledge" of the speed of the flywheel is embodied in the structure of the device itself. So, while it is possible to describe the Watts governor as having "knowledge" about some rules of behaviour, it is actually misleading to do so. Your understanding of the device will be much deeper if you realise that the apparent rule-governed behaviour is an emergent property of the way the device is built. This is the level at which I think we have "knowledge" of syntax, etc.

      A couple of points about my over-stating the reliance of linguists on definitions: I don't mean to say that linguists sit there consulting dictionaries and trying to define words. I'm just pointing out that there is a very strong belief in our culture that words have stable core meanings and I think that part of this belief is related to the fact that our culture has dictionaries, which are not compulsory artefacts of language use. So, the last thing you say - that language can be studied without invoking dictionary definitions - is precisely my point. Yet, we are very hung up on the concept of core meaning (which is a parallel to the concept of a definition). This belief has a strong influence on the types of problems we try to solve in linguistics and psychology.

  8. Sabrina - You first state that a linguistic stimulus caused by a language speaker is just another auditory stimulus. Using your definition of "information" as I understand it, that means that the stimulus has time-varying structure from which information can be extracted for use in determining a responsive action. Such actions are context-dependent and are learned (eg, from teachers or by trial-and-error). So, there appears to be no difference in principle between a language-capable organism responding to auditory stimulus in that language and any organism responding to any auditory stimulus.

    But then you introduce a distinction between auditory stimulus due to the word "dog" and auditory stimulus due to a referent of that word (possibly not presently perceivable), attributing additional "linguistic information" to the former. This seems to unnecessarily confuse the issue since it isn't clear to me what role that additional "information" plays in an "ecological analysis" as I understand that term. (Of course, I'm not denying that language is useful in the ways you enumerate, but that seems a separable issue.)

    I also find the example involving a word and a referent of the word confusing. So, let's consider instead two language speakers, one who utters the sentence "Do you have a dog?" and one who utters the sentence "Haben Sie einen Hund?", both to a monolingual (say, English) hearer. While both stimuli are auditory and each is linguistic from the POV of its utterer, from the POV of the hearer only the one in English is linguistic; the one in German is - in terms of linguistic "information" - just auditory noise. But we can assume that the hearer has learned at least one response to each sentence that is context-appropriate; perhaps the response to the English utterer is "Yes", to the German utterer some gesture(s) signifying failure to understand. Then, two stimulus-response transactions have been executed based only on auditory "information". To address any supposed linguistic information associated with the word "dog" alone would seem to take one down the representational path that was to be avoided. I think this view can be extended to more complex exchanges, but for the moment I just wonder if you disagree with it even in this simple example.

    (The response to the German speaker is clearly not "about" anything, and it isn't clear to me that the response to the English speaker is either. In any event, if you mean "about" in the sense of intentionality ala Brentano, that seems an issue - possibly avoidable - that is relevant only to the structure of a language.)

    Re affordances: on reviewing yet again some of Andrew's previous posts thereon, I noticed points (previously missed or under appreciated by me) re obligatory effecting. I think I had not taken his emphasis on context-dependence sufficiently seriously as a possible resolution to the multiple simultaneous affordances problem. If you include every aspect of context in determining whether an immediately effectualizable (right word?) affordance is available and look at effecting such affordances as a continuous process, it seems quite reasonable (though not necessarily correct) to assume that one and only one must exist at any instant. (I'm not conversant in dynamical systems, but my gut feeling is that this might amount to assuming something about the continuity of the equations defining the dynamics of the system and their derivatives). Similarly, as a strict determinist, I have to assume that when all aspects of context are included in determining the response to a linguistic stimulation, only one is possible. So, while I agree that "If some one tells me 'pick up the red one'" I don't necessarily have to comply, I don't agree that whether you do comply or not is optional.

  9. Hi Charles,

    Response to the first half:

    I've been working hard to sort out consistent terminology, but I'm sure I've made some errors. My current idea is to refer to auditory events when simply talking about acoustics in the auditory array, to refer to perceptual information when that event is about the thing in the world that caused the sound, and to refer to linguistic information when that event is about something else. I agree that from the first person perspective linguistic and perceptual auditory events aren't particularly different because in both cases the task of the organism is to figure out how to use the information appropriately.

    So, why am I making this distinction? 1) Because from the third person perspective they are different in an important way (their relation to what they are about) and this difference means that only perceptual information can be used in the continuous control of action and only linguistic information can tell us about things that aren't currently perceivable. In order for this account of language to be at all rigorous, I think it's important to recognize that perceptual information is special because of how it is related to the physical world. If I refer to all auditory events as perceptual information, or if I abandon the information idea and only refer to auditory events, then this important distinction risks being lost. 2) The second reason I want to use separate terms is because humans are uniquely adept at using linguistic information. Keeping the terminology separate reminds me that this needs explaining - it's not just that language is a particularly difficult perceptual problem and I want to know what properties (of humans and of language) make language special.

    The German example you provide doesn't trouble me. As you point out, a non-German speaking person who hears a sentence in German has not learned how to respond appropriately to that information, but he will have learned how to respond to sentences uttered in his own language. In neither case though does this person think that the auditory event is about the thing that caused it - the physical production of speech sounds. Even when hearing a sentence that he doesn't understand, he knows that the sound is about something else - something conventional.

    I agree that the word "about" in my account is wishy-washy. I'm still working at how to refer to meaning and the use of "about" is a not very subtle attempt to avoid the word "meaning" while I sort this out. Of course, using "about" doesn't buy me anything, but it might be somewhat less likely to trigger assumptions of core word meaning.

    I want to be able to explicate the trade off between stability and variability in what language seemingly refers to. In perception-action, what an information variable is about is well-defined, even if its particular meaning at a given moment depends on a multidimensional system including other perceptual information. I can look at tau and know exactly what properties of the environment it tells me about. Similarly, I can look at stability in a person's actions and know something about the kind of task specific device they are. But, my ability to do this depends on defining tasks with respect to the information variables they involve. Linguistic information doesn't have this relationship to the world, so there is no obvious starting point for accounting for stability in word use. I could come up with some plausible sounding stories about stability in the environment and learning and what-not, but I think this would be a cop out. I want a better explanation.

    1. I think I see the distinction you want to make. We can't respond to an object or event that we can't perceive, eg, that isn't present in both space and time (Brentano's "inexistent entities"). You see language as bridging that gap, and so do I. But I wouldn't describe the distinction in terms of what the perceptual event itself is "about". If I ask you "Who is the PM of the UK?", the perceived event is my utterance of the question, which you perceive as being an English sentence. Now, the structure of that particular sentence considered as an analyzable linguistic event is such that it is indeed about the inexistent entity "the PM of the UK" (unless David Cameron" happens to be present at the time). But I assume that the response to a simple perceptual event like that is latent in your memory and is triggered directly and immediately by the perceptual event itself, not as the result of further analysis of the perceived event as also being a linguistic event. If this is correct, the linguistic structure of the sentence is irrelevant. Perceived sentences for which latent responses are not found or which are complex may require further processing as linguistic events, but this would be true whether or not they are about inexistent entities. In short, while I agree that linguistic events in general are different from mere perceptual events, my guess is that a lot of linguistic events are processed just as perceptual events. And I don't think "aboutness" distinguishes those categories.

      I'm sympathetic with your desire to avoid the word "meaning" by replacing it with some phrase about the response that the stimulus is intended to evoke. But if that proves too cumbersome, you could do the same thing that you're trying to do with "information" - define "meaning" carefully in those terms and apply the word consistently.

      I don't quite get your stability concerns, so I'll have to think more about that issue.

    2. I see what you're getting at here. I agree that from the first person there is not necessarily any additional processing of a speech event as linguistic. In this sense the fact that it is speech is irrelevant. I still think it's important to make the distinction though. One additional reason for this that I didn't mention before is that the third person differences between perception and language (specification versus convention) shape the hypotheses we will generate about learning and in the tools available to study language acquisition. There *is* something special about language in the sense that humans are the only animals to have developed it. The specialness might reside in the evolutionary function of language for our species (i.e., it fills a need for us that doesn't exist in other species), in our ability to learn it (brain size, cognitive resources), or in our desire to communicate via symbols. I don't know. But, treating language as identical to perception doesn't recognise our unique use of language. In recognition of this, I have chosen to highlight one clear difference in the relationship between perceptual information and linguistic information. This might turn out to be uninteresting and something else entirely might account for the fact that humans developed language while other animals didn't. But, it's worth keeping on the table until we know better what we're dealing with.

    3. Have you read any of Quine's work on language? I'm reading Peter Hylton's book "Quine", and on rereading Chapter V - which addresses Quine's views on language acquisition - just noticed that it seems directly relevant to your project.

      In case you haven't, some highlights from the chapter. Quine sought to explain the "aboutness" of language (in the broad sense sense of reference, not the narrow sense of intentionality ala Brentano). He defines "observation sentences", which are a sort of primitive on which more complex language is built, and distinguishes what he calls "proto-observation sentences" which are initially learned by children as essentially unstructured wholes that are handled perceptually rather than cognitively. (Sounds familiar, no?) Anyway, his views seem somewhat aligned with yours, so even if it turns out you're going down a wrong path, you seem to be going down it in outstanding company!

    4. Thanks for reminding me about Quine. I read some of his stuff years and years ago and remember being impressed, but he had fallen off my radar. The observation sentence idea seems possibly related to Tomasello's construction grammar, which I've encountered in Everett's new book. The point is that sentence structures can be learned just like individual words and that hitching these structures together in various ways creates linguistic behaviour that appears as if it requires a detailed knowledge of syntax, when, in fact, it does not.

    5. Yes, that idea is prominent in Hylton's discussion of Quine's work. Although they (he and Quine) apparently see the range of sentences amenable to "perception-only" processing in producing a response much more limited than I do - perhaps due to their immersion in the high intellectual content discourse of academia. But a lot of public discourse - even on substantive topics - seems to me intended to produce not reflective but reflexive response.

    6. I just recalled this Quine paper which seems to capture the essence of his "observation sentences" in few pages. Sentences he calls "holophrastic" are processed as entities, ie, in a perception- response mode. He briefly discusses how primitive versions are learned by children and even complex ones emerge with frequent use, eg, in specialized fields with their own vocabularies.

  10. I have a question I hope you'll take the time to answer. Possibly it's a bit naive as I'm a student still trying to get to grips with all this stuff, but here goes.

    How does writing work without representations? As in a fiction writer, dreaming up worlds, people and situations in his or her head before expressing part of it in words by typing it out. What is in a writer's head if not a representation of a story?

    Or how does reading a novel work without representations? I'm halfway through a book right now and I can remember the plot, characters, etc. without remembering a single individual word or sentence I read. What is my memory of a narrative if not a representation of a story?


    1. It's a good question, and it's an important issue that we are very keen to tackle.

      One issue is that just because there is persistence and dynamic stability within us and our nervous systems doesn't mean that that stability is best described as a representation.

      This isn't just about terminology though, although precision there is part of what we're up to generally and that Sabrina is pushing very hard here. There are some very real problems with the concept of representation; a selection of posts on this topic

      Chemero on theories of representation in which he tries to define representation in a general way that is fair to the topic but is a theoretical construct worth rejecting.

      A selection of problems with representation, where Sabrina goes through this in some detail.

      What else could it be?, where Sabrina discusses the Watts steam governor, a great metaphor for non-representational cognitive science.

      So yes: there is clearly some kind of internal structure and stability in the nervous system and we use that structure to do a lot of things. But it is not compulsory to call that structure a representation, and there are many reasons described in the posts above why we might not want to call it a representation.

      I'll leave the language specific stuff to Sabrina :)

    2. Thanks for your question. Andrew covered most of what I would have said, but I'll make a couple of additional points. First, representations have a specific theoretical meaning, but psychologists tend to use the word very loosely to refer to anything that we have the ability to remember or conjure up in our heads. The most general form of "representation" is an internal mediating state. That is, stuff happens in the head than influences the relationship between a stimulus and response. The most common additional requirement of representations is that they must be discrete rather than continuous (Dietrich and Markman explain at length why, but mostly it rests on the idea that representations need to be transformed via cognitive processes).

      I don't dispute that people learn over time and that they have the subjective experience of being aware of things that they recall or things that they invent. These behaviours and experiences are representational only if they meet the requirements of being representations - that is, they must be discrete entities (states) and they must be transformed via cognitive operations that are, themselves, functional representations. In other words, invoking representations necessitates thinking about cognition as "information processing" - the good old computer metaphor.

      This metaphor has dominated the discipline for decades, so it is bound up in the way we talk about pretty much everything to do with cognition. It also dovetails nicely with our intuitions about our experiences (like the examples you provide about writing). So, what's the problem?

      1) In practice, psychologists invoke the word "representation" to mean all sorts of things. This is just poor science. There is also no consistent theory about how representations are linked to neural activity. The original cognitive psychologists were explicitly against linking the two, but most modern cognitive psychologists assume that representations must somehow be identical to patterns of neural activity. 2) Although some tasks show evidence of internal mediation of behaviour (language is often involved), there is not any good evidence that this internal mediation takes the form of representations - that is, as discrete entities that are modified by operations. Pretty much anything that can be described in these terms can also be described in terms of dynamical systems. Dynamical systems are non-representational and (usually) continuous. The problem is that psychology hasn't given dynamical descriptions of cognition a fair shake. Again, this is bad science. 3) Non-representational accounts of cognition are counter-intuitive, but that doesn't make them less plausible than representational accounts. Quantum physics doesn't make any sense at all but it is unreasonably well-supported by data. 4) We have a good theory (ecological psychology), which can be a foundation for a new way to think about the problems that need to be solved by cognition. I've tried to do this here with an ecological discussion of language. It isn't a perfect account, but if I'm being disciplined about my use of theory, then the most logical conclusion I can draw right now is that language use is non-representational. Now, of course, I need to collect some data!!

    3. Ok, thanks both for replying. I'm not sure how representative the Dietrich and Markman view on the need for discrete representations really is, as it's not that widely cited, but I can totally see why you can argue that those kinds of representations aren't necessary. They sound like old-school amodal symbolic representations.

      But I guess I'm more familiar with the Barsalou-style simulation account of representation, which is definitely not discrete. It even looks to me like these's not much difference between the distributed simulation view and the dynamic systems view... except the former calls them representations and the latter doesn't. Maybe, as you say, it's more a matter of tightening up the terminology!

    4. The reason that I cite the Dietrich and Markman paper is that they are very knowledgeable about the theoretical underpinnings of representations, even though I think they are wrong about the necessity of them. I used to really like the Barsalou approach - my PhD supervisor knew him and we were all supporters of his perceptual symbols systems. However, Barsalou doesn't understand perception. He has a very old-school understanding of how it work and this means he thinks the brain has certain work to do (related to poverty of stimulus) that it really does not have to do. The other issue with Barsalou is that he still thinks symbols are needed for cognition, even if these are rooted in perception (he was having it out with the abstract symbol system people, he wasn't anti information processing). So, even though he thinks the content of representations is perceptually-based, he still thinks it is necessary to have something "stand in for" the world that we apply mental operations to. Actually, the Dietrich and Markman view is essentially identical to this because they acknowledge that many representations will be based in perceptual experience, so all of the arguments they present will be valid for Barsalou's view as well. So, the view that Andrew and I present is fundamentally different than Barsalou's. If you haven't taken a look at the "What else could it be" post that Andrew linked to, then check it out to see how the computational and dynamic accounts differ (or, just check out the Van Gelder paper that it cites).

    5. Anonymous,
      At the least we should be willing to accept that the involvement of representations in these processes is a hypothesis. That is, there are things that people do, and past researchers (psychologists, philosophers, linguists, etc.) have assumed that the best way to explain these 'things that people do' is via a hypothesized representational system (with some level of dualism thus entailed).

      But there is now increasing evidence, at least in part due to increasing sophistication in understanding physical processes, which suggests that a representational system is simply unnecessary. In fact, there is plenty of evidence that a few hundred years pursuing psychology-minus-representation will be a lot more productive then the past few hundred years pursuing the representation hypothesis.

  11. Can you comment on the "Louder than words" by Benjamin K. Bergen? How close is it to the approach to language and embodied cognition you are describing in your blog? Thank you.

  12. Calling into the void: Hello, I am reading this 8 years later. It is my intro to eco psych and I am fascinated. I've read the two posts on language through an eco-psych lens and the debates in the comments about whether representation as conventionally understood is necessary for language processing. I'm wondering if anyone could link some updated material on this subject, or just offer a 2020 take? Has new information weighed in more heavily on one side than the other?