On the nature and perception of depictions

This online forum is devoted to the perception of depictions. In this essay, I discuss several issues that I believe are relevant. My discussion is focused on film and on film theory. However, I believe that my arguments apply to all forms of depiction, that is, to visual depictions (e.g., drawings, paintings, sculpture, silent movies, visual virtual environments), to auditory depictions (e.g., radio), and to depictions that are available to multiple perceptual modalities (e.g., cinema, television, and multimodal virtual environments). As these examples indicate, my concept of depiction is broad. I am interested in the nature of depictions (that is, what distinguishes depictions from things that are not depictions). I am also interested in how depictions influence the structure of potential sensory stimulation (that is, the structure that depictions impart to ambient light, sound, and so on, and how these structures differ from structures imparted by non-depictions). Finally, I am interested in the perception of depictions: What do we perceive in the presence of depictions? Is perception of depictions accurate? Can we distinguish depictions of things and events from corresponding real things and events, and if so, how? In my discussion, I take the perspective of the Ecological Approach to Perception and Action (e.g., James Gibson, 1979/1986). In this essay, I combine elements from my own earlier writings on film theory (Stoffregen, 1997), on the nature of simulation (Stoffregen, Bardy, Smart, & Pagulayan, 2003), and on general theories of perception (Stoffregen & Bardy, 2001).

Depictions are of interest to many cognitive and perceptual psychologists. This interest is theoretical (e.g., this online forum) but it is also practical. The practical issue arises because there is a venerable and robust tradition of using depictions as laboratory stimuli in experimental research on perception and cognition. In most cases, the researchers conducting these studies seek to understand perception and/or cognition, in general. That is, there is a long-standing assumption that perception and/or cognition in the presence of depictions is equivalent to (or even identical with) perception and/or cognition in the absence of depictions, that is, when we are confronted only with “real” things. I believe that this assumption is not valid.

The perception of depictions

Are we fooled by film?

Films are widely believed to give rise to “the illusion of reality”. For example, in presenting a novel theory of film Anderson (1996, p. 161) claimed that the perception of films is illusory, or non veridical. He attempted to explain the putative illusion of reality by an hypothetical capacity of the visual system “to process a synthetic array of light as reality,” (p. 113). James Gibson made similar statements, suggesting that the viewer has “an awareness of being in the place and situation depicted”, (1979/1986, p. 295). In these statements both Anderson and Gibson imply that the film is perceived as real (i. e., that viewers believe they are in the presence of physical events, rather than depictions of physical events). Gibson went so far as to assert that “the seated viewer never actually turns his head, of course, but he gets the essential optical information for doing so”, (p. 298), implying that when the camera pans the viewer might perceive his or her head to turn (cf. Hochberg, 1986). At the same time, however, these authors realize that viewers are not fooled. Gibson was aware that “no one is ever wholly deceived”, (J. J. Gibson, 1979/1986, p. 301), while Anderson pointed out the simple fact that “we do not get up and run out of the theater at the appearance of the monster”, (Anderson, 1996, p. 47). Gibson concluded, “the illusion of reality is a myth” (1979/1986, p. 281). These statements appear to be contradictory. If the synthetic array is processed as reality, and if it can contain information specifying a head turn, then how can it be that viewers differentiate filmed events from physical ones?

The nature of depictions

Films are a subset of depictions: In perceiving a film we perceive a depiction. One of the few psychologists to grapple with the nature of depictions, as such, was James Gibson, who returned to this issue repeatedly over several decades (for a summary statement, see Gibson, 1986/1979). What is a depiction? After struggling with this seemingly simple question for many years, proposing and rejecting a series of candidate definitions, James Gibson (1979/1986, p. 271) defined a depiction as a surface treated to provide “an array of persisting invariants of structure that are nameless and formless”. Gibson proposed that in structuring light depictions create a “duality of ... information”, which gives rise to “a dual experience”, in which the things that are depicted “are not perceived, and yet they are perceived”, (1979/1986, p. 283; cf. Gibson, 1971). This derives from his claim that “a picture is both a surface in its own right and a display of information about something else” (1979/1986, p. 283). However, this does not eliminate the contradiction, leading Gibson to argue that the perception of depictions is “a paradox” (1979/1986, p. 282). Similarly, Anderson describes film as creating “two incompatible sets of information” (p. 48).

Gibson attempted to embrace this contradiction by concluding that “a picture always requires two kinds of apprehension that go on at the same time, one direct and one indirect”, (1979/1986, p. 283), with the perception of the surface being direct and the perception of the subject of the depiction being indirect. I think that this strategy deals with only part of the problem. An implication of “incompatible”, “dual”, and “paradox” is that the perception of the surface and of the content of the depiction are mutually exclusive, and/or that the depiction structures light in two ways that are mutually incompatible. In this both Anderson and Gibson may be mistaken. If the optic array is a product of physical law, then its structure cannot bear a paradoxical relation to physical reality, any more than physical law can produce a paradox. Similarly, if films are created in accordance with physical law, then they cannot structure optic arrays in ways that are incompatible or paradoxical. The subject of the depiction may be paradoxical (e.g., the lithographs of M. C. Escher), but the specification cannot be.

I think that the problem arises from Gibson’s assertion that “a picture is a surface that always specifies something other than what it is”, (1979/1986, p. 273; see also Gibson, 1971, p. 32). The paradox would vanish if depictions were specific to only a single, unitary thing. I believe this to be the case. Rather than simultaneously specifying something that is there (the surface) and something that is not there (the subject of the depiction), the array structure engendered by a depiction specifies only one thing: It specifies a depiction. To perceive a depiction, then, is to have a unitary perception of an image on a surface, or, more properly, a state or event depicted on a surface. Rather than directly perceiving the surface and indirectly perceiving the subject of the depiction, I am suggesting that we have a single, direct percept of the depiction, as such. There is no need to postulate paradoxical specification, or paradoxical experience. There is no need to propose distinct percepts of “that which is depicted” and “that which supports the depiction”. To perceive a depiction is not to perceive “an image, and a surface”, but “an image on a surface”.

My definition of depiction may help to resolve the muddle over whether viewers perceive films (or other depictions) to be real. No matter how compelling the depiction, we never run from the monster. This is easily explained when we understand that in movie theaters people do not have separate percepts of that which is depicted and the surface on which it is depicted. To perceive the movie monster is, fundamentally, to perceive that the monster is in the movie. The depiction is perceived, as such. My position is implicit in Gibson’s assertion that “the notion of an image that is literally and actually indistinguishable from the reality is a myth”, (1971, p. 33). However, Gibson did not discuss this point in the argument that led him to conclude that the perception of depictions is paradoxical (for example, compare the section entitled “The naive attitude and the perspective attitude”; 1971, p. 32, with the section entitled “But what about the illusion of reality?”; 1971, p. 32-33).

The specification of depiction, as such

I regard it as a fact of fundamental importance that viewers differentiate depictions from “real” things, and from “live” events. In addition to not running from the movie monster, we behave very differently when we observe a film of a murder than we would if we were present while a murder was being committed. Similarly, we behave differently in the presence of a volcanic eruption than we do when the eruption is depicted in a movie. Surely this is so because we perceive the depictions, as such; that is, we differentiate the meaning of depictions and “real” events. What is the basis for this differentiation?

In an ecological theory if depictions are perceived, as such, then they must be specified, as such. Specification occurs when there is a unique, 1:1 relation between some aspect of reality and a pattern of ambient energy (J. J. Gibson, 1979/1986; Stoffregen & Bardy, 2001). When this unique relation exists between reality and ambient energy, then the patterns in ambient energy are sufficient for accurate knowledge, that is, a perceiver who is sensitive to the patterns in ambient energy can, on that basis alone, perceive the world accurately. Perception of this kind is said to be direct, as opposed to indirect perception, which is required if patterns in ambient energy bear a non-unique or ambiguous relation to reality. When stimulation is ambiguous, accurate knowledge of the world requires intervening mental processes that remove ambiguity from the stimulus. The Ecological Approach to Perception and Action asserts that specification exists, and that perception generally is direct (Gibson; Stoffregen & Bardy).

How is depiction specified? The answer is that depictions (e.g. a film of a baseball game) and “real” things (e.g., a baseball game) give rise to different structures in ambient arrays (for optics, a long and diverse list of differences is given by Hochberg, 1986). A depiction, no matter how technologically sophisticated, cannot have perfect fidelity to that which is depicted (J. J. Gibson, 1971, p. 33; 1979/1986, p. 279-280). One reason for this is that the depiction is finite (in space, time, and/or resolution), while the structure in the undepicted environment is unlimited. Depictions and depicted differ also with respect to changes in the point of observation. Changes in the position or motion of the point of observation give rise to variants and invariants of array structure that differ for depictions and depicted. For example, movements of an observer’s head when viewing a baseball game will give rise to different optical changes than when viewing a film of a baseball game. Another difference is that viewing of films (and pictures) is partially open-loop (not responsive to perceptual information), whereas viewing of the depicted events (if we are really in front of them) is wholly closed-loop, or controlled with reference to perceptual information (cf. Smets, 1995, p. 192). These differences probably underlay much of the ease with which depictions are distinguished from depicted. The difference between open-loop and closed-loop viewing is so salient that even suckling infants show acute sensitivity to it. For example, Kalnins & Bruner (1973) presented infants aged 5-12 weeks with visual displays whose focus was contingent on the infants’ sucking rate. When sucking improved focus, it was rapid and prolonged. When sucking reduced focus, it was suppressed. Ironically, the stimuli used by Kalnins & Bruner were films; their findings should, therefore, be of direct relevance to Anderson’s (1996) thesis.

If depictions are specified, as such, and perceived, as such, then the issue is not how, or even whether, film is perceived “as reality”. Film is not perceived as reality, just as a radio drama, a stage play, a painting, or the performance of a mime are not perceived as reality. All of these depictions are perceived for what they are. There is no illusion of reality because there is no illusion, at all. This does not mean that viewers (or listeners) necessarily perceive the depicted events to be false in the sense of being fundamentally fictional. It is possible, and common, to perceive the depicted events to be real. This can occur even when they are fictional, as with Orson Wells’ notorious radio broadcast of War of the worlds. But while many listeners believed the events in Wells’ production to be real, no one believed that the events were taking place in their radio sets, or in their living rooms. In this sense the depictions were perceived, as such.

The analysis in this section and the previous one requires us to develop a new explanation for the compellingness of film. Films are perceived as depictions, rather than as “the real thing”. A film may “seem real”, but this does not mean that we believe that it is real (that there is no depiction). Given this, how is it that we find films to be so absorbing? It may be that, rather than being drawn into the film (the depiction, as such), viewers are drawn into the story, that is, into the events that took place in front of the camera (this is what happened with Wells’ War of the worlds). The real question, then, would be how it is that film (or radio, or a stage play), seems to be able to draw us into the story (or the diegesis, to use the formal term). How is it that we can become so involved in things that we clearly perceive to be depictions? This issue of involvement (as opposed to perception) leads us back to relations between film and other media, such as literature. Readers know that involvement with a good book can be just as intense as with a film. I am not suggesting that the experience of books and films is identical; obviously, it is not. What I am suggesting is that an understanding of film, per se, will require us to understand relations between film and other story-telling media (some early experiments on this issue were conducted by J. J. Gibson and R. M. Gagne; Reed & Jones, 1982, p. 17).

Multimodal perception of uni-modal depictions

In discussing how filmed and “live” events structure ambient arrays Anderson (1996), Gibson (1979/1986), and Hochberg (1986) all limit their analysis to optics. Yet observation of film (and of depictions, in general) structures multiple forms of stimulus energy (Stoffregen & Bardy, 2001; Stoffregen, Bardy, Smart, & Pagulayan, 2003). Exploratory movements (e.g., of the eyes and head) cause changes in structure of gravitoinertial force, strain in muscles and joints, and pressures on the skin, as well as changes in the optic array. Consider a film that depicts rotation of the point of observation (e.g., a pan, which bears some resemblance to a head turn). Physical turning of the head influences the optical structure that is sampled, but it also influences the stimulation of the vestibular and somatosensory systems. Thus, a presentation that is limited to the optical consequences of a head turn (i.e., a film) does not specify that the head has turned. To put it more precisely, a real head turn leads to changes in the orientation or position of the head relative to the visible environment, relative to the audible environment, relative to the environment of forces (e.g., gravitoinertial force), and relative to the rest of the body. The head turn produces changes in patterns of optical, acoustic, gravitoinertial, and mechanical stimulation (that is, changes that can be seen, heard, and felt). By contrast, a pan (the filmic approximation of a head turn) leads to changes in the orientation and position of the head relative to portions of the visible environment, and relative to portions of the audible environment, but not relative to the environment of forces, or relative to the rest of the body. Thus, considered jointly, multisensory stimulation provides abundant perceptual information that makes it possible, in principle, for observers to differentiate head turns from camera pans. Countless laboratory experiments have demonstrated that people are acutely sensitive to variations in their orientation relative to the gravitoinertial force environment (e.g., leaning, tilting, or rotating), and that we are acutely sensitive to changes in the orientation of parts of the body relative to one another (Clark & Horch, 1986). Thus, we have good reason to believe that people routinely distinguish camera pans from head turns. This is likely to be part of the reason that viewers know that they are looking at a film, and not at the corresponding reality.

Film can be a uni-sensory art form (think of silent movies), but the perception of film is never confined to a single perceptual system. Viewing of film (or of paintings, or listening to radio, etc.) always involves stimulation of multiple perceptual systems. This is because perception is an act (it is something that we do, rather than something that happens to us). The act of perception depends upon movement of the perceiver. In viewing visual imagery (paintings, or film) perception entails movements of the eyes in the head, movement of the head on the neck, movement of the trunk relative to the legs and (for standing viewers) movement of the entire body.

Conclusion

The issue of how film is perceived has implications that extend beyond the entertainment industry. Issues of film perception are relevant to recent attempts to understand teleoperation (Smets, 1995), and to the dynamic depiction of industrial processes (Vicente & Rasmussen, 1992). Film perception may also have considerable relevance for scientific psychology (Hochberg, 1986). This is because much of contemporary psychological research depends on a methodology in which humans are asked to perceive depictions. The favored screen is that of a computer monitor (E. J. Gibson, 1991, p. 607; Reed, 1996), but the issues of depiction, and the perception of depictions, are much the same for computer displays as for film. Such studies have an uncertain relation to the perception of physical events. Contemporary experimental psychology may tell us more about how we perceive depictions than about how we perceive the world in which we evolved. It is important that we understand what the perception of depictions is, but it may be important in a much broader sense that we understand what the perception of depictions is not (cf. Ittelson, 1996; Reed & Jones, 1982, p. 16-17). This can be done only if we conduct research that does not depend on depictions. It would also be extremely useful to conduct direct comparisons between perception and action relative to depictions and relative to physical surfaces, objects, and events. Consistent with one aspect of James Gibson’s analysis (1971, p. 33), I have concluded that films can be distinguished from physical events, and that this is so because the difference is specified. This may have important implications for research that utilizes depictions. It may also suggest logical (as opposed to technological) limitations on the fidelity of simulation (Stoffregen et al., 2003; cf. Edgar & Bex, 1995).

Acknowledgments

Preparation of this essay was supported by Enactive Interfaces, a network of excellence (IST contract #002114) of the Commission of the European Community.

References

Anderson, J. D. (1996). The reality of illusion: An ecological approach to cognitive film theory. Carbondale, IL: Southern Illinois University Press.

Clark, F. J., & Horch, K. W. (1986). Kinesthesia. In K. R. Boff, L. Kaufman, and J. P. Thomas (Eds.), Handbook of perception and human performance: Vol. 1. Sensory processes and perception (pp. 13-1 – 13-62). New York: Wiley.

Edgar, G. K., & Bex, P. J. (1995). Vision and displays. In K. Carr and R. England (Eds.), Simulated and virtual realities (pp. 85-102). Bristol, PA: Taylor and Francis, Inc.

Gibson, E. J. (1991). An odyssey in learning and perception. Cambridge, MA: MIT Press.

Gibson, J. J. (1971). The information available in pictures. Leonardo, 4, 27-35.

Gibson, J. J. (1986). The ecological approach to visual perception. Mahwah, NJ: Lawrence Erlbaum Associates, Inc. [Original work published in 1979]

Hochberg, J. (1986). Representation of motion and space in video and cinematic displays. In K. R. Boff, L. Kaufman, and J. P. Thomas (Eds.), Handbook of perception and human performance (Vol. 1, chapter 22). New York: John Wiley and Sons.

Ittelson, W. H. (1996). Visual perception of markings. Psychonomic Bulletin & Review, 3, 171-187.

Kalnins, I. V., & Bruner, J. S. (1973). The coordination of visual observation and instrumental behavior in early infancy. Perception, 2, 307-314.

Reed, E. S. (1996). Encountering the world. New York: Oxford University Press.

Reed, E. S., & Jones, R. Reasons for realism: Selected essays of James J. Gibson. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Shapiro, M. A. & McDonald, D. G. (1992). I’m not a real doctor, but I play one in virtual reality: Implications of virtual reality for judgments about reality. Journal of Communication, 42, 94-114.

Smets, G. J. F. (1995). Designing for telepresence: The Delft virtual window. In P. Hancock, J. Flach, J. Caird, and K. Vicente (Eds.), Local applications of the ecological approach to human-machine systems (pp. 182-207). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Stoffregen, T. A. (1997). Filming the world: An essay review of Anderson’s The reality of illusion. Ecological Psychology, 9, 161-177.

Stoffregen, T. A., & Bardy, B. G. (2001). On specification and the senses. Behavioral and Brain Sciences, 24, 195-261.

Stoffregen, T. A., Bardy, B. G. Smart, L. J., & Pagulayan, R. J. (2003). On the nature and evaluation of fidelity in virtual environments. In L. J. Hettinger and M. W. Haas (Eds.), Virtual and adaptive environments: Applications, Implications, and Human Performance Issues (pp. 111-128). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Vicente, K. & Rasmussen, J. (1992). Ecological interface design: Theoretical foundations. IEEE Transactions on Systems, Man, and Cybernetics, SMC-22, 589-606.