The Ecological Approach to Virtual Reality

As virtual reality (VR) gear gets better, cheaper, and easier to use, there is renewed interest in trying to figure out how best to make a virtual environment feel real.  The typical for framing for VR is in the name: it's about creating the illusion of reality, a virtual world. Programmers even talk this way; they describe creating virtual (pretend) environments, objects, spaces, etc. From this point of view, VR is an attempt to create a stable illusory experience by faking a world that doesn't really exist. 

Of course, VR programmers don't make worlds; they make information. This makes folding VR into the ecological approach a natural move, and I propose that ecologically, VR development is actually an attempt to design an optic array containing information that can support certain behaviours. It's less virtual reality, and more virtual information. This is important because the nature of the information we are using explains the form of the behaviour we are controlling. Your goal as a developer is therefore not to create tricks and illusions, but to provide information capable of supporting the behaviours you want to be possible,

As a first step towards an ecological understanding of VR, I will first follow the path Gibson laid down taking the science of perception away from illusions and towards information. I'll then think about some of the implications of taking an ecological approach for VR design. Virtual reality needs our theory of perception to become the best it can possibly be, and I hope that this post serves as an entry point for designers to become aware of what we have to offer them.

Illusions, Pictures, Mirrors and Gibson

de Wit, van der Kamp & Withagen (2015) describe in nice detail how Gibson approached the question of visual illusions (I blogged the paper here). The key thing is how Gibson connected his studies of illusions to his studies of the perception of pictures. He notes

A picture is a surface so treated that a delimited optic array to a point of observation is made available that contains the same kind of information that is found in the ambient optic arrays of an ordinary environment. (Gibson, 1971, pg 31)

So for Gibson, 'seeing the illusion' is actually 'correctly detecting the information put in front of you' (see also Turvey et al, 1981, on 'Misperception, Misconstrued'). Typically, that information is lawfully created by light interacting with real dynamical properties. Pictures are a crude way of presenting that information without needing the properties present - they are a kind of VR device. They are simple and limited because they are static and also because they create information that they are a surface and not the thing depicted, but they can sometimes get you some of the way. Video can get you further, and 3D movies a little further than that, although all these are limited because they optic array you see does not change as you move.

Virtual reality is simply the latest technological upgrade in this ongoing fascination with creating surfaces that structure the optic array so as to look other than they are. For most VR systems, that surface is a computer or projection screen, and the optic array it creates is designed to make that surface look like something other than it is (e.g. a virtual environment). The thing that makes VR a qualitative shift in making optic arrays is that it uses surfaces that can change how they are structuring light in response to the way the perceiver is moving. Pictures reveal themselves to be pictures when you move relative to them and their "I'm a picture" properties become apparent in the optic array. With VR, suddenly, it's possible to continuously conceal the fact that there's an intermediate surface by never letting it create information that it's there.

Mirrors do this, actually. As I review here, we are terrible at perceiving that there is a surface there; instead, we perceive what the optic array depicted on the surface specifies, namely a mirror-reversed version of the optic array in front of the mirror. This is because this 'virtual' array has excellent fidelity and responsiveness to our movement. Mirrors are so good at this that some researchers use them to create virtual environments for people to interact with, with great success (e.g. Coats et al, 2008). Computer powered VR is limited by the quality of the algorithms and the power of the computer, but it's big advantage is of course the fact that you have more and easier control over the contents of the virtual array.

What Counts as Realistic?

This article is interested in a major question in VR; how can we make the virtual environments realistic and immersive, given the constraints the systems must work with? The focus is, of course, on the illusions and tricks programmers deploy, and many of these are very interesting. People are also excited as computing power increases, making photo levels of realism possible.

Ecologically, however, the answer is different: the virtual optic array is experienced as realistic and immersive to the extent that it contains ecological information that can support the coordination and control of the actions people want to do in the specified environment. Photo-realism is not the goal - info-realism is. Rob Gray makes this point in great detail in his excellent three part podcast series on VR in sports research and training, here, here and here.

What makes information realistic? I propose that information is experienced as if it were real if that information is stable and supports functional interaction with the dynamic it is supposedly about. In other words, it feels real if it lets you successfully do something in particular. Part of this is about processing speed; people are very susceptible to lags between moving and optic flow, and these can make people feel very ill. But once the hardware has hit a certain point, more speed is not the trick to realism; appropriate informational content is.

Ecological researchers have been at the forefront of VR research for years, because we need to be able to manipulate information in order to study it's effects and sometimes those manipulations are not physically possible. Geoff Bingham has been study prehension in VR for many years, and Bill Warren has a large room people can walk and run in that he uses to study the visual control of locomotion and interception. None of these experimental set ups are photo-realistic. In fact, they are often quite stripped down. But they always contain all the relevant ecological information to support the task being studied; that is, in fact, always the point of these experiments. For example, Warren studied the outfielder problem and VR let him do impossible things to the ball trajectory to compare OAC and LOT strategies. The displays were very plain; essentially simple textured surfaces, a simple glove and the ball. But I got to play in it on a visit many years ago and it was quite amazing; in a few minutes I was happily running around, trying to catch virtual balls and utterly immersed. Everything I needed to intercept a ball was in my optic array, so it was realistic in the most important way - it could support my behaviour.

Another recent fun example; Disney recently were showcasing how they could use tricks to enhance skilled interactions in a virtual environment. They asked people to catch real balls being tracked and displayed in the headset, and added various components to the display to try and help people do this. the best part, as I blogged here, is that they didn't need to do anything! Because their initial displays contained accurate trajectory information about the real ball, people were quickly catching around 95% of the balls. Their 'enhancements' was a prediction generated by the computer about where the ball would arrive, but this forced people to switch from the stable, flexible and effective prospective control method to a less stable, jerky predictive mode. The prediction technically added information to the display, and people could learn to use it, but it was not ecological nor did it support smooth, stable control. Prediction may be the engineer's solution, but evolution is not an engineer.

Virtual Reality Programmers Need to Understand Perception

Design is best when it doesn't fight the nature of the person using it, but when it works with you (to paraphrase Donald Norman, if you have to label it you built it wrong). If you want people to push on your door, don't put a handle on it. A virtual environment is a designed product, specifically a designed optic array that aims to be able to support quite complex, continuous, real time behaviours. This means that the design should try to work with the perceptual systems that support such behaviours.

Ecologically, behaviour takes shape as a function of the nature of the information being used to select, coordinate and control actions. In the non-designed world, the dynamics of the world interact with energy arrays and, via the local laws of physics, create kinematic patterns that can specify those dynamics (see here for more). Organisms then learn to use those patterns as information for the dynamics that created the pattern (law-based use) or as information for something else (convention-based use; Golonka, 2015). Information is only created when a dynamic property is present and interacting with an energy medium; this means predictive control is never an option and supporting prospective control should be the goal (this is why Disney engineers adding prediction information messed with people so much). 

In the designed world, those kinematic patterns can be simulated in a couple of ways. First, they can be calculated directly as kinematics, as motions. The equation would specify where the object should be at each instant, given certain parameters about how it was being viewed or moved. Second, they can be the outcome of simulating a dynamic; the computer would run a differential equation and the object would behave according to that.

The latter, while harder, is a better solution - simulating the dynamics is a more robust, flexible solution while it's easy to break a system trying to directly compute a position every frame. We are very sensitive to things that don't move right; a good movie can be ruined because the kinematics of the CGI didn't quite map any coherent dynamic. Generating motions from dynamics also scales better than directly generating the motions; swarms of CGI animals or characters are created dynamically these days (think the orc armies in Lord of the Rings).

Summary

Virtual reality involves the design of an optic array that can support the selection, coordination and control of fairly complex, continuous and time-extended interactions with the environment. Because it is a designed product, there is the possibility of both good and bad design. Good design occurs when the designed product enables the user to interact with it smoothly and efficiently, and that is promoted when the designer works with, rather than against, the capabilities of the user. Good VR design therefore needs to understand the ecological nature of perception and action.

VR design is a perfect example of an intrinsically multi-disciplinary project. At this point, few if any people have all the necessary technical skills and scientific knowledge to do it alone. The future of VR is therefore in teams, of programmers, hardware specialists, sensor designers, and ecological psychologists.