The medium for direct perception (Notes on Van Dijk & Kiverstein, 2020)

The ecological approach has hit a point in its history where it has become interested in expanding its scope, to go beyond the real time coordination and control of action. There are many challenges from non-ecological cognitive science about how to tackle representation-hungry problems, and how to conceptualise things like language, social behaviour, and what the brain is up to. I am all on board with this move - it was important we waited till we were ready, but since Gibson died in 1979, the empirical programme on the basics has matured into a solid foundation and we have a lot of developed or adopted a lot of things that will come in useful. 

However, if we are going to do it, I want us to do it with rigour and care and with reference to all our hard-won successes. My current view is that our best path lies in looking at the ways we are able to use ecological information, and grounding our explanations and hypotheses at this scale. Sabrina first developed this idea in a paper about how to think about what language is (Golonka, 2015). The big take-home from that paper is the analytic distinction between law-based use and convention-based use of information, and the first draft of the consequences of this distinction. We built on this when we started thinking about brains (Golonka & Wilson, 2019), and I'm currently thinking about the next step along this path. 

I'm pretty sure that a big chunk of the work I need to do is explicitly connecting this distinction up to work on the skilled intentionality framework, and the notion of our variable levels of grip on the field of affordances. This work is wrong about affordances (they aren't relations) but other than that, there's a ton of really great work about how intentionality isn't an all-or-nothing thing, and a lot of really useful vocabulary and framing development that I think will be useful for articulating these ideas. I don't like re-inventing wheels, so I'm skilling up on this literature as I develop ideas for a paper. 

This post is about a recent paper (Van Dijk & Kiverstein, 2020) that is explicitly about developing a usage-based notion of information. To unbury the lede, I think this is a robust piece of work with solid internal logic, but I think like all this enactivist style work, it ends up in a place that cannot support a how-actually explanation of behaviour - this particular usage-based theory of information and the things that come with it aren't the framework that will let the ecological approach expand its scope. This is ok, at one level, because I don't think mechanisms are the goal of enactivist analyses. But it's a worry at another level, because I want an ecological theory of direct perception that can actually explain behaviours and this isn't going to cut it. 

The goal of this paper is to develop a usage-based account of ecological information that fits within the radical empiricist tradition (which Gibson was firmly a part of). The problem they are trying to solve is the nature of the 'medium' for direct perception, which they argue has been developed by Gibson in two ways, each incompatible with each other and one incompatible with radical empiricism. The overall work of the paper is to develop a single view of the medium that can then be applied to explain how all kinds of patterns can come to serve as information when used. 

Two Views of the Medium for Direct Perception

Gibson developed his idea of the medium from Heider's account, which distinguished between 'thing and medium'. A thing is something that is relatively solid, resists being deformed, and persists in its form over time. A medium is something that is relatively fluid, and is capable of being shaped and formed by what it interacts with. 

Gibson's first sense of the medium is as a medium-for-perception. Take the optic array. This can be a medium for perception because the forms that it takes can be lawfully established by the things it interacts with, and because it is ambient to both the organism and its environment. The form of the optic array changes over time in a way that the form of things cannot (optic flow) and those changes are available to be detected. This sense of the medium, then, is as a 'ready made' structure: the forms of the various perceptual arrays are created ahead of being detected, and are there to be discovered. 

Gibson's second sense of the medium is as a medium-for-action. Some parts of the physical world resist actions taking place in or through them; they are substances ('things') or surfaces (where substances and media meet). Action is possible, however, because there are media that do not completely resist organisms, things like air and water. Actions can flow through a medium the way information flows through a medium. How these work as media is not ready-made, however; they become media in negotiation with an organism's actions. For example, water can serve as substance for something that cannot swim, but can come to serve as a medium if that organism learns to swim. The medium now is not ready-made, but defined relative to the activity and the organism. 

Van Dijk & Kiverstein make two moves at this point. First, they note that these two senses are mutually incompatible, and that only the usage-based sense of the medium aligns with radical empiricism. Second, they then go on to develop a usage-based account of the medium for perception, in the form of a usage-based account of information, in order to get all senses of the medium under one, appropriate notion; this is the rest of the paper. 

(I will note at this point that I'm not convinced there is a problem with these two senses; it may simply be the case that the medium-for-information and the medium-for-action work differently, and that this will be ok with clear accounting of how the word is ever being used. However, making perception and action effectively the same kinds of things that actually exist as perception-action is a solid ecological move, so this may end up being important). 

Use Is What Makes Structure Information

The current ecological approach to information is that it consists of higher-order patterns in ambient energy arrays, that these patterns are the law-based projection of world dynamics into those energy arrays, and that the law-based projection allows the patterns to be specific (map 1:1) to those dynamics. Specification, considered this way, makes it so that the pattern is about the dynamics; it is information, and this comes ready-made for the organism. 

Van Dijk & Kiverstein note that this is the problematic sense of the medium. They are happy that there are indeed higher-order patterns in ambient arrays, but they deny that these can be considered as information until they are used by an organism. They will make this distinction throughout the paper by talking about patterns vs information-relations. 

Specification as a Process

They now need to deal with a consequences of this move. In the standard approach, what makes it possible to say a pattern is about something is specification. The standard, Turvey notion is that specification lives between the surface and the pattern in the medium and is underwritten by the law-based projection of the former into the latter. This makes patterns in the medium-for-perception information about things ahead of use, and is a key part of the Turvey approach to making direct perception possible; specification is required for information to allow but not mediate perception of the world. 

Van Dijk & Kiverstein want to jettison all this, so they need an equivalent way to get aboutness into their story, without losing the directness of perception and while maintaining their radical empiricist understanding of perception as a process. Their claim is that specification, the thing that makes something information about something, is underwritten not by a law but by the activity of the organism. Specification is a process, and as such is not all-or-nothing; there can be varying amounts of specification and it gets better as the activity of the organism becomes more and more successfully coordinated with the demands of the environment. They claim that this notion supports everything Turvey-specification supports, gets it to align with radical empiricism, and naturally allows for a discussion of social experience.

(This is a big, big claim, which they defend in the next section. I will note at this point that I think calling all this 'specification' is just an error; this just isn't what that word means, and the resulting work is pointed in the wrong direction.)

A couple of consequences: first, there is no ontological distinction between specifying and non-specifying information in this account; because specification is negotiated into being as the organism-environment fit improves with actions, these aren't different in kind. Second, issues such as normativity (was the action appropriate to the circumstances?) are defined pragmatically; if it works, it was good, and if something worked better, it was better. Van Dijk & Kiverstein pitch both of these as advantages, part of getting all the ecological ducks into a radical empiricist row. 

How All This Enables Direct Perception

In this view, direct perception is not possible because of pre-given specifying, meaningful information about the world. Instead, it has to be possible as the result of skilled activity;

Specification...is an outcome that takes form over the course of ongoing situated activity. Perceiving is the process of achieving an information-relation. In this ongoing process, the active animal explores for and uses the patterning available in the ambient array so as to establish an information-relation. The animal achieves coordination by using ambient patterns in the activities of 'listening, touching, smelling, tasting, and looking'

Then how you've done this in the past and how you are doing it now maintain and develop the specification-in-action. 

I have a lot of questions. When this is still in the early stages of developing, is there still specification because the process is going on? Or is there not yet specification, because it's still early days and the animal-organism fit isn't optimal yet? If the latter, is the perception that's happening still direct? How? And how does all this get going? How do I get into this perception-action loop of developing specification in action if none of the available patterns mean anything? Which patterns do I try to use, and why? Why do I change which patterns I'm using, and how? I'm sure there are answers possible with this framework, so I don't mean these as slam dunk problems; but I would like some answers, my answers drag this right back into Turvey-land, and this paper does what most of these accounts do and starts with the up-and-running trained system. 

The Social World as a Medium

Van Dijk & Withagen notes that the standard view of information as specifying variables struggles to explain how the social world can constrain our behaviour (this bit made me wonder if it's actually true and write this Twitter thread in which I draft the idea that it's not actually a problem). The worry is that social practices aren't obviously specified in ambient arrays, but from their account this is ok, if we treat sociomaterial practices as a medium in which human activities unfold. (One issue here; we don't perceive a medium, we perceive in them. So how does a medium per se shape our behaviour?)

The idea here is that the constraints of the social world are what make it so that the material things that we engage with have social meaning. For example, when I interact with an alarm clock in order to wake up at a certain time so as to get to work on time, what makes that interaction mean all that is the socio-cultural medium in which alarm clocks play certain roles and I have deadlines that align with the activities of other people, who are all also working within the same medium. When we do this right, the medium acts as a medium and we all arrive in the same place at the same time for our meeting. When we do this wrong, these things all create resistance and trouble; the analogy is how water shifts from a medium for locomotion to a substance that impedes locomotion and vice versa. 

Effectively, the meaning of everything is still always being negotiated into existence, and in socio-cultural contexts, that negotiation occurs in the medium of the social world and it's conventions. This is again supposed to be the same kind of thing as negotiating the meaning of higher-order patterns in ambient arrays, which occurs in the medium for perception and it's laws and conventions. 

All of This is Direct Perception

The final section is intended to justify the claim that this usage notion of information and the medium still allows direct perception. The argument takes the form of an extended analysis of enacting a given affordance that is initially distant in space and time (the affordance of catching a train; whether this is actually an affordance is another question, but the argument is premised on the idea that everything a person is doing as they work towards catching the train is organised with respect to catching the train). 

Direct perception is what emerges over the activities, each of which is done as part of catching the train and done in a way to reflect that goal (e.g. you walk to the train station, and you do that walk in a way to get you there on time for the train). Everything unfolds over time and converges on the actual act of stepping onto the train; specification of the train catching affordance is achieved, not relied on. 

Effectively, this is still just an interesting way to describe the phenomenology of doing things to catch a train, but it still doesn't really explain how it all works. 

My Thoughts

I think this paper is well-motivated. It is attempting to do more nitty-gritty work on how it can be possible that behaviour can be organised with respect to a spatially and temporally distant goal (such as catching a train) without invoking mental models/representations. It tries to do this by deconstructing some key parts of the ecological analysis of direct perception, then reconstructing them within a single, more coherent, radical empiricist framework. The authors identify that direct perception needs a concept of the medium (the context within which meaning is established), and opt for one in which meaning is never pre-given but only ever negotiated by the activity of the organism in the medium. This radical empiricists concept of the medium can be applied to mediums for perception (such as the optic array), mediums for action (such as water) and mediums for culture. Patterns in any of these media can come to mean things via skilled action, and so behaviour in all these media is the same kind of thing. 

Like all this enactivist style work, we end up with a nice clear description of what's going on, but no clear reason to accept this description over any other. Take the discussion of catching the train - the word 'anticipation' is doing a lot of work that could readily be done by 'prediction'. In order to create an argument in favour of the former over the latter, we're owed an account of how this actually works, how this actually plays out. This paper doesn't provide that. This is ok - perhaps a future paper will do it using this one as a guide. In one sense, this paper is decomposing a theory that can generate explanations of behaviour, reconceptualising some of the resulting parts and processes while working to keep all the necessary properties, and then resituating the new parts into a theory they claim can do the same work as the old one, plus some new work, all within a single coherent approach. 

But I have reason to think trying to create a how-actually explanation of this framework will fall over. Take specification. Van Dijk & Kiverstein clearly recognise that specification is required somewhere to support the directness of perception, but their analysis doesn't let them localise specification to the lawful relation between a surface and a pattern in an ambient array. They have to localise it elsewhere in the system, specifically in the process of perception-action. But this doesn't work; that process isn't up to implementing specification and you end up with 'varying amounts of specification', which is just not what specification means and isn't the kind of thing that can do what specification needs to be able to do in order to support direct perception. Ending up with parts and processes that can't implement properties required of the system is a hint that the scheme guiding the analytic decomposition of the system is flawed. 

This paper is a solid piece of work; the internal logic holds and the goal is noble. But it ends up with the same problem created by rejecting traditional notions of affordances (as dispositions) and information (as patterns in ambient arrays that specify those affordances) - it is entirely unclear how to develop this how-possibly verbal description of behaviour into a how-actually explanation of behaviour and there are many reasons to think it won't work.