The Ecological Approach to Virtual Reality

As virtual reality (VR) gear gets better, cheaper, and easier to use, there is renewed interest in trying to figure out how best to make a virtual environment feel real.  The typical for framing for VR is in the name: it's about creating the illusion of reality, a virtual world. Programmers even talk this way; they describe creating virtual (pretend) environments, objects, spaces, etc. From this point of view, VR is an attempt to create a stable illusory experience by faking a world that doesn't really exist. 

Of course, VR programmers don't make worlds; they make information. This makes folding VR into the ecological approach a natural move, and I propose that ecologically, VR development is actually an attempt to design an optic array containing information that can support certain behaviours. It's less virtual reality, and more virtual information. This is important because the nature of the information we are using explains the form of the behaviour we are controlling. Your goal as a developer is therefore not to create tricks and illusions, but to provide information capable of supporting the behaviours you want to be possible,

As a first step towards an ecological understanding of VR, I will first follow the path Gibson laid down taking the science of perception away from illusions and towards information. I'll then think about some of the implications of taking an ecological approach for VR design. Virtual reality needs our theory of perception to become the best it can possibly be, and I hope that this post serves as an entry point for designers to become aware of what we have to offer them.

The Affordances of Natural vs Designed Environments (A Purple Peril)

I had a conversation with Eric Brymer (@Ericbrymer) a few weeks ago. Eric is a Reader at Leeds Beckett and is an ecologically minded psychologist interested in the effects of the natural world on mental health. We chatted affordances for a while, and despite it being a very interesting chat, I really wasn’t sure if I had anything to say about the differences between the natural and designed worlds. 

But I have not been able to stop thinking about this topic, and now I think there’s something fairly cool here.

Information is typically dense and continuous (A Purple Peril)

Optic flow is everywhere, all the time (same with other energy arrays, like the acoustic array). We depend on this fact deeply. When we are cast adrift from information, our behaviour quickly accumulates errors and strays in often disastrous ways. One example is the case of friction, which doesn't exist until two surfaces are in contact and therefore does not create information about itself that is available ahead of that contact. In another example, when cut off from landmarks, people walk in huge circles, getting seriously lost and confused; Souman, Frissen, Sreenivasa & Ernst, 2009). A simple version of this is the game of walking with your eyes closed; you quickly lose all confidence about where you are and what's happening and it's actually very difficult to make yourself walk at normal speed. 

The Perilous proposal is that behaviour emerges in real time, as a function of the current flow of information, and that this flow is typically dense and continuous, not intermittent. I will illustrate this with an example of two designed sets of instructions for navigating though a building, where the dense information set leads to better, more stable behaviour.

Embodiment and design; the affordances of pedestrian crossings

I'm a sucker for good design. I'm interested in things that work well because they are designed with the right task in mind. Take the greatest potato masher of all time, the Spudnik. It works well because it mashes as the result of a very natural action with the arm, easier and less effortful than the more traditional device. Good design works with the user and the task (rather than trying to impose a behavior) because behaviours that are supported by the task and the environment will be stable, reliable and easy to maintain. 

Design is interesting for embodied cognition because it's an attempt to artificially manipulate the environment to create affordances for some but not other behaviours. Sometimes it's just a matter of getting the physical layout right (see the Spudnik). Some of these required behaviours, however, are quite complex and not the sort of thing that would typically create information if left to their own devices. A great example is the design of streets to promote safe driving and pedestrian behaviour; a lot of the rules being designed for are human conventions, not physical necessities, and so without someone intervening and building something there couldn't be perceptual information anywhere about that rule. In addition, the designed element often creates information about something other than itself (see the Aboutness dimension in Sabrina's information taxonomy). This in turn creates the possibility for there to be more than one way to create an environment that produces relevant information and can therefore shape behaviour, and in turn, this allows the possibility that some designs will be better than others.

With this in mind, let me introduce you to two examples of design that I would like to change; the staggered pedestrian crossing, and puffin crossings. Both of these artificially restrict access to useful information in ways that mean well but that I think fight too much against human behaviour. I actually started preparing a grant to empirically investigate these designs from a more embodied perspective, and the EPSRC thought it was in their ballpark. More pressing concerns intervened, but I would love to actually do these studies and would like to hear from anyone who might be interested in collaborating ("Dear Pamela lab..."). I think our embodied cognition approach (Wilson & Golonka, 2013), with it's focus on task analyses and information, could really have an impact on an interesting part of our day to day life.

Breaking the (ecological) law: why illusory sounds don't make for safer cars

Electric cars are great for the environment, but they come with a problem - they're too quiet, and this makes them dangerous to pedestrians used to the loud noises of the internal combustion engine. One idea is to add noise to the cars, and Mark Changizi recently wondered whether perceptual psychologists could help design a better sound, using an illusion. I don't think it would work, and the reason is a nice example of why it's important to understand the relationship between the world and perceptual information about the world.

Events in the world create information. When a car moves towards you, for example, there are sounds coming from the engine, and the way these sounds change over time is information about the heading of the car. If it's getting louder over time, it's coming towards you; but if the pitch is decreasing at the same time, then the car is going to pass you, not hit you. Increasing volume and constant pitch, however, specifies a collision and the particular rate of change of these variables tells you about the time-to-contact. These patterns of change over time in the acoustic array are related to the way the car is moving via the laws of physics and are therefore informative about the details of the car's motion. Importantly, the relationship between these patterns is also governed by the laws of physics; increasing volume and constant pitch only go together when the car is really heading towards you, for example.

Every perceptual system has limits, though, and those limits are called thresholds. When a stimulus is outside a threshold (e.g. too quiet, or too high pitched) the perceptual system won't respond to it.  The problem with electric cars is that they are quiet at low speeds, and so the variation in pitch you need to detect a collision is small and possibly below threshold. You could make the cars louder, but this goes against one of the selling points of the cars - reduced noise pollution. Mark suggested creating an artificial sound, an illusion, in which the range of the variation in pitch created by the slowly moving car is amplified without just making the car louder. This is in principle possible, but it's a problem because it's breaking the law, and you end up with less information about the car than when you started.