1 Multisensory Virtual Worlds

Enactive knowledge depends upon an action – perception cycle. Action reveals information, which guides further action, which reveals additional information, and so on. To be successful, action must be controlled. One aspect of control is stability; precise perception and performance depend upon the stable control of movement. This raises the question of the referents for stability; stable relative to what? It also raises questions about perceptual support for action: What perceptual information is required for stable control of movement relative to appropriate referents? 

We discuss the role of multisensory enactive technologies in supporting the perception and control of the user’s actions. We argue that users must simultaneously control aspects of their behavior relative to the virtual and real environments. Thus users need to have perceptual information about their motion relative to both real and virtual referents. Information about these differential motions exists in higher-order, emergent patterns that extend across different forms of stimulus energy (e.g., light, sound, forces), and is not available in patterns within individual forms of energy. We have argued (Stoffregen & Bardy, 2001) that perception and action are inherently multisensory, and that their multisensory nature is irreducible. Here, we argue that enactive interfaces can be enhanced if designers take into account information that exists in emergent, higher-order properties of multisensory stimulation. 

2 Control in Virtual Worlds

2.1 Nested Actions

Consider a standing person using a head-mounted display in which a head-tracker and a data glove are used to update the display in real time. Upright posture must be maintained. Walking may be an option. The user must control gaze, and position of the hands. These tasks are interdependent. The position of the virtual hand, for example, is influenced by movements of the real hand, but also by movements of the real arm, and by movements of the real body in walking, and in body sway. Similarly, the direction of gaze in the virtual environment is influenced by movements of the head, torso, and limbs. In general, whole-body movements, such as walking and body sway, have “ripple effects” on control of body segments, such as the arm, hand, and head. Reciprocally, movements of perceptual and effector systems (e.g., eyes, hands), impose constraints on the control of the limbs and torso (Riccio & Stoffregen, 1988; Stoffregen et al., 1999). 

2.2 Relative motion and referents for movement

We move. Perception involves motions of receptor systems (often including the whole body), and action involves motion of effectors (often including the whole body). Thus, the perception and control of behavior is largely equivalent to the perception and control of motion. Movements are controlled and stabilized relative to some referents. To watch tennis, the eyes must be stabilized relative to the moving ball. To stand, the body must be stabilized relative to the gravito-inertial force environment. Action and perception can be controlled relative to a myriad of referents. 

2.3 Multiple, Simultaneous Referents

Animals control different aspects of their behavior relative to different referents (Riccio, 1995). A person moving relative to one referent but in stasis relative to another, for example, may simultaneously control their orientation and motion relative to both. Consider driving at constant velocity. During turns, as the direction of balance changes relative to the surface of the earth, the torso remains aligned with the direction of balance (that is, it rotates as the direction of balance rotates), but the head and eyes may maintain their orientation relative to the road (Figure 1). A similar effect occurs in flight (Figure 2). During turns, the pilot must control the orientation (and position) of the aircraft relative to the surface of the earth (for navigation), while aircraft orientation must be maintained relative to the direction of balance (for aerodynamic stability). In general, perception and control relative to multiple, simultaneous referents is adaptive (Riccio, 1995). 

Figure 1. We tend to align the torso with the direction of balance (DOB), and the head relative to the road. During linear travel on a flat road (A), these two referents have a fixed relation to each other. But during turns (B), the DOB varies independent of the road, with the effect that the torso and head may be oriented differently.

Figure 2. In aviation, the body tends to remain aligned with the direction of balance (arrow), while the head may be aligned with the interior of the aircraft, or with the ground, depending on the task. 

We must select referents for the control of action. The selection of referents should have a functional basis (Riccio, 1995), that is, it should depend on the goals of action (e.g., a pilot who controls orientation relative to the ground may lose aerodynamic control, and a pilot who controls navigation relative to gravito-inertial force will get lost). One aspect of learning to perform new tasks will be the determination of which referents are relevant. 

3 The Global Array

3.1 Direct perception of physical referents

In daily life, we control our actions relative to various aspects of the environment, such as objects, surfaces, vehicles, and people. How does perception operate to facilitate these actions? In this section, we describe two theories of perception; the classical theory (indirect perception), and the ecological theory (direct perception). Classically, perception and motor control are assumed to depend upon internal referents, such as the retina and cochlea. These internal, psychological referents for the description and control of motion are known as sensory reference frames. Sensory reference frames are necessary if sensory stimulation is ambiguous (i.e., impoverished) with respect to external reality; in this case, our position and motion relative to the physical world cannot be perceived directly, but can only be derived indirectly from motion relative to sensory reference frames. Motion relative to sensory reference frames often differs from motion relative to physical reference frames (e.g., if the eye is moving relative to the external environment). For this reason, sensory reference frames bear only an indirect relation to physical reference frames. If perception and control are based on sensory reference frames, then accuracy depends on the perceiver’s ability to “fill in the gaps” between motion defined relative to sensory reference frames and motion defined relative to physical reference frames. Traditionally, this “filling in” is assumed to require inferential cognitive processing. 

The idea that sensory stimulation bears an ambiguous relation to physical reality is an assumption, and it may be wrong. We claim that sensory stimulation is lawfully determined in such a way that there exists a 1:1 correspondence between patterns of sensory stimulation and the underlying aspects of physical reality (Gibson, 1979; Stoffregen & Bardy, 2001). That is, we assume that reality is specified in available sensory stimulation. If this is true, then perception can be direct, in which case, behavior can be perceived and controlled relative to physical referents; sensory reference frames are not necessary.

3.2 A super-ordinate array

The concept of ambient arrays was developed in the context of single forms of energy, and there is wide acceptance of the existence of (at least) the optic array (Gibson, 1966) and the acoustic array (Stoffregen & Pittenger, 1995; cf. Gaver, 1993). Ambient arrays in individual forms of energy are subordinate components of a higher-order entity that we call the global array (Stoffregen & Bardy, 2001). The global array consists of spatio-temporal structure that extends across multiple forms of ambient energy. 

The global array is structured by all events, objects, and surfaces that structure single-energy arrays. In addition, the global array is structured by events that do not structure single-energy arrays, including differential motion relative to distinct physical referents. Some examples illustrate the existence of information in patterns that extend across forms of energy. These examples focus on patterns that extend across two or three forms of energy. Our discussion in terms of a limited number of forms of ambient energy is for clarity of presentation only. 

Figure 3. Single-energy arrays (A) and the global array (B) as a vehicle slows to a stop. A.U.: Arbitrary units.

Consider a situation in which an automobile slows to a stop. Figure 3a shows the consequences of this motion for structure in the optic array and in the gravito-inertial array. Optical structure is ambiguous with respect to motion relative to the gravito-inertial environment: The same optical patterns could be caused by deceleration of the body relative to the ground, or by deceleration of an illuminated enclosure relative to a gravito-inertially stationary observer. At the same time, gravito-inertial structure is ambiguous with respect to the nature of the motion: The same patterns of acceleration could be caused by deceleration to a stop, or by acceleration (in the opposite direction) to a constant non-zero velocity. Figure 3b shows the higher-order relation between optics and gravito-inertial force that exists in the global array. This “optical-gravito-inertial pattern” specifies that the observer is undergoing gravito-inertial deceleration relative to the illuminated environment. An animal that was sensitive to this higher-order pattern would be able to perceive its motion directly.

Structure in the global array can be formalized. Bingham and Stassen (1994) identified information about the distance of illuminated objects from the observer. The head and torso move forward and back during ordinary body sway. These movements change stimulation of the visual, vestibular, and somatosensory systems. The optical parameter t (the inverse of the relative rate of dilation of a contour in the optic array) is influenced by the physical time-to-contact, Tc, of the head with the distal object or surface. However, Bingham and Stassen noted that optical flow created by head motion is ambiguous with respect to distance unless there is independent information about the velocity of head motion. Head movement structures gravito-inertial patterns that are available to the vestibular system. Thus, the higher-order relation between head velocity and optical flow is unambiguously related to object distance:

tpv/T = (1/2p) (D/A), (Eq. 1)

where tpv is the value of t at the peak velocity of head motion, T is the period of head oscillation, D the target distance, and A head movement amplitude. This information is used for the perception of distance in virtual environments (Bingham & Pagano, 1998; Bingham et al., 2000; Panerai et al., 2002; Wickelgren et al, 2000). 

To catch an object, we need information about when and where it will arrive. Peper et al. (1994) identified a parameter in the global array that provides information about the velocity of hand motion needed for interception:

Vh = (Xb - Xh) / t, (Eq. 2)

where Vh is the hand velocity necessary for catching, Xb is the object’s instantaneous sideward position, Xh is the hand’s current position, and t is the object’s Tc with the body’s fronto-parallel plane. Optical structure is influenced by Xb and t, while mechanical stimulation is influenced by Xh, and so the right side of Equation 2 constitutes a pattern in the global array.

The global array is not equivalent to the idea that a given piece of information may be detected by more than one perceptual system, such as hearing a musical rhythm or seeing the same rhythm by watching the performer. This idea, that information can be redundant across modalities, is also known as amodal specification. Intersensory redundancy exists, but does not consist of or imply the higher order, superordinate patterns that consitute the global array. For an extended discussion of this issue, see Stoffregen & Bardy (2001, Section 3.3.

3.3 Simulation of Global Array Patterns

The global array provides information that can optimize perception and performance, and (we claim) this information is not available in any other form of sensory stimulation. Humans may detect informative global array patterns, and they may routinely use this information for perception and control, in both VE and daily life. These assertions place a premium on the availability of relevant global array patterns to users of virtual worlds.

Because it is an emergent property of the various forms of ambient energy, the global array does not need to be created. Rather, it is an inescapable consequence of the existence of multisensory stimulation. Thus, VE designers do not need to make special efforts to make the global array available to users: The global array already is available to users. Rather than attempting to create the global array, designers need to become aware of the global array that already exists, and begin to understand how multisensory displays structure the global array. 

Consider research on the multisensory control of perception and action in virtual environments. This work is built on the identification by Bingham and Stassen (1994) of a global array pattern that provides information about the distance of objects (Equation 1). By manipulating the properties of single-energy displays, Bingham simultaneously manipulated the emergent global array parameter that provides information about ego-centric distance. Bingham has shown that VE users can detect this parameter of the global array (Bingham & Pagano, 1998; Bingham et al., 2000; Panerai et al., 2002; Wickelgren et al, 2000). For us, the essential aspect of this research program is the initial identification of the relevant global array parameter (Bingham & Stassen). This made it possible to construct laboratory situations in which this parameter could be manipulated, and in which its perceptual salience and utility for performance in virtual environments could be evaluated. 

4 Applications of the Global Array in Enactive Virtual Worlds

If ordinary perception and control rely on the global array, then virtual worlds that capitalize on this will have greater verisimilitude, better subjective fidelity, faster learning, and better task performance. These benefits should accrue for any virtual world that includes self-controlled movement of the user.

We should begin by identifying the physical referents relative to which various aspects of behavior should be controlled for a given task. This will inform us about constraints on the control of action and will indicate the dimensions of the global array that contain information for control. We should next identify and formalize global array structures that provide information about observer motion relative to these referents. Next, multisensory enactive interface systems can be designed to make this information accessible for users. Always, we need to remember that action is controlled simultaneously relative to referents in both real and virtual worlds. For this reason, relevant patterns in the global array will include but extend beyond multisensory enactive interfaces, and perceptual exploration and pickup will include but extend beyond interfaces. 

References

Bingham, G.P. & Pagano, C.C. (1998). The necessity of a perception/action approach t definite distance perception: Monocular distance perception to guide reaching. Journal of Experimental Psychology: Human Perception and Performance , 24, 145-168.

Bingham, G. P. & Stassen, M. G. (1994) Monocular egocentric distance information generated by head movement. Ecological Psychology, 6, 219-238.

Bingham, G.P., Zaal, F., Robin, D. & Shull, J.A. (2000). Distortions in definite distance and shape perception as measured by reaching without and with haptic feedback. Journal of Experimental Psychology: Human Perception and Performance, 26, 1436-1460.

Gaver, W. W. (1993). What in the world do we hear? An ecological approach to auditory event perception. Ecological Psychology, 5, 1-30.

Gibson, J. J. (1986). The ecological approach to visual perception. Mahwah, NJ: Lawrence Erlbaum Associates, Inc. [Original work published in 1979]

Panerai, F., Cornilleau-Peres, V., & Droulez, J. (2002). Contribution of extraretinal signals to the scaling of object distance during self-motion. Perception & Psychophysics, 64, 717-731.

Peper, L., Bootsma, R. J., Mestre, D. R. & Bakker, F. C. (1994) Catching balls: How to get the hand to the right place at the right time. Journal of Experimental Psychology: Human Perception & Performance, 20, 591-612.

Riccio, G. E. (1995) Coordination of postural control and vehicular control: Implications for multimodal perception and simulation of self-motion. In: Local applications of the ecologial approach to human-machine systems, P. Hancock, J. Flach, J. Caird & K. Vicente (Eds.). Lawrence Erlbaum Associates, Inc.

Riccio, G. E., & Stoffregen, T. A. (1991). An ecological theory of motion sickness and postural instability. Ecological Psychology, 3, 195-240.

Smart, L. J., Stoffregen, T. A., & Bardy, B. G. (2002). Visually-induced motion sickness predicted by postural instability. Human Factors, in press. 

Stoffregen, T. A., & Pittenger, J. B. (1995). Human echolocation as a basic form of perception and action. Ecological Psychology, 7, 181-216.

Stoffregen, T. A., & Bardy, B. G. (2001). On specification and the senses. Behavioral and Brain Sciences, 24, 195-261.

Stoffregen, T. A., Bardy, B. G. Smart, L. J., & Pagulayan, R. J. (2002). On the nature and evaluation of fidelity in virtual environments. In L. J. Hettinger and M. W. Haas (Eds.), Psychological issues in the design and use of virtual and adaptive environments. Mahwah, NJ: Lawrence Erlbaum Associates, Inc. In press.

Stoffregen, T. A., Hettinger, L. J., Haas, M. W., Roe, M., & Smart, L. J. (2000). Postural instability and motion sickness in a fixed-base flight simulator. Human Factors, 42, 458-469.

Stoffregen, T. A., & Smart, L. J. (1998). Postural instability precedes motion sickness. Brain Research Bulletin, 47, 437-448.

Wickelgren, E. A., McConnell, D. S., & Bingham, G. P. (2000). Reaching meaures of monocular distance perception: Forward versus side-to-side head movements and haptic feedback. Perception & Psychophysics, 65, 1051-1059.