1. Introduction

Scientists use images frequently. Visual representations like graphs, diagrams, MRIs and electron micrographs appear in research reports, grant proposals, textbooks, lab meetings and lectures. The way scientific images are presented indicates that they are not merely objects of perception, but are also involved in scientific reasoning. Figures in published research reports, for example, contribute to the argument presented in the paper: sometimes by expressing its conclusion, and sometimes by providing evidential support. How are we to understand reasoning that involves figures?

One way to address the issue is to investigate the use of images in light of current ideas about cognition. The depictive character of mental models provides a perspective on human cognition from which to clarify why visual representations are so useful to scientists. This tactic provides a perspective on the images that goes beyond individual cases, but is not constrained to explain the reasoning involved solely in terms of properties of the images. I will give a brief explanation of mental models and then examine several different kinds of cases, in order to show how images are involved in scientific reasoning.

2. Mental Models

One general hypothesis about cognition is that mental representations may have a depictive character rather than the features of a linguistic representation. External linguistic representations are characterized by a serial format: the sequence of visible marks or sounds is sufficient to determine a meaningful sentence. Spatial relations among the marks, such as distance from one word to another, do not normally contribute to linguistic meaning. In contrast, a depictive external representation is one whose two-dimensional form is essential to its meaning: some spatial relations of the symbol refer to features of the referent. In scientific visual representations, the spatial relations of the figure might refer to non-spatial or even non-visible properties. For example, length in timelines represents duration, and graphs are used to represent relations among properties. For all visual representations, however, the relationship between symbol form and content correlates visible properties of representations with features of the states of affairs they represent.

Recent work in cognitive science suggests that mental representations have similar features. A mental model is a depictive mental representation, in the sense that it has the same structure of relations as their contents. This means that dimensionality of content will be reflected in the dimensional structure of the mental representation. This is similar to how external visual representations use spatial dimensions to represent spatial and temporal relations, and other dimensional properties like temperature. If internal representations are structured like their referents, then manipulation of structural elements of the internal representation that are shared with the referent will serve as a form of reasoning, because the forms of both the input and the output representations are correlated with the structures of their referents.

Thinking about mental representations as mental models contrasts with the view of mental representations as analogous to linguistic representations. If mental representations have a serial format rather than a dimensional, depictive format, then the structural form of mental representations would stand in an arbitrary relation to the content represented, just as the visible form of the marks comprising a printed sentence bears an arbitrary relation to the content expressed by the sentence. In this case mental processes would run like operations on serial symbols. Rules of inference could be applied based on the form of a mental representation in a manner analogous to applying rules of inference based on the form of logical formulae, but not by altering or changing perspective on the form, as would be the case with mental models.

While the question about the nature of mental representation is not yet settled, there is a growing body of evidence in support of mental models. Problem-solving through mental imagery suggests that some cognitive processes are best described as manipulations of mental models rather than operations on serially structured mental content. For example, when we use visual imagery to think through a problem like counting the sides of a pyramid, we use a mental representation that shares some of the structural features of the object we are thinking about. Experimental results in which the amount of time subjects take to solve problems correlates positively with some dimension relevant to the problem (like an estimation of distance or shift in perspective) provide further support. Cooper and Shepard (1973), for example, test time required to rotate letters and other patterns.

3. Images & construction of mental models

The concept of mental models is very general. Can it provide any insight into scientists’ use of figures? Understanding mental representations as mental models provides some criteria for what must be accomplished in comprehending an external representation. No matter how the external representation is formatted, its content must be given a depictive structure for the internal representation. This suggests that images provide an advantage, because the structural relation between external representation and referent may facilitate formation of an internal representation which also has a structural relation to the referent. The information figures convey is already formatted so that spatial dimensions map onto relevant parameters of the state of affairs represented. Any symbol must be perceived in order to be comprehended, but because visible features of the symbol are mapped onto features of the referent, visual representations are comprehended through a perception of the visible features of the symbol which have a built-in structural relation to the symbol’s content. The cognitive convenience that visual representations offer would then provide an explanation of why figures are used when alternative forms of representation are available. The presentation of content in two-dimensional format provides a way to decrease the total effort to needed to form an internal representation with a spatial character.

Figure 1: Diagram of the binding change mechanism of the ATP synthase. Jan Pieter Abrahams, A.G. Leslie, R. Lutter, and J. Walker (1994), “Structure at 2.8 A resolution of F1-ATPase from bovine heart mitochondria.” Nature, 370 6491:621-8. reprinted with permission from Nature; www.nature.com.

Consider an image that conveys a scientific model. Figure 1 is a diagram of the mechanism by which an enzyme complex called the ATP synthase catalyzes the formation of ATP—the compound called the “energy currency of the cell” because it provides the immediate energy for most cellular functions. Most ATP is formed in mitochondria as the end result of a multi-step biochemical process called oxidative phosphorylation. In the last step of oxidative phosphorylation, the ATP synthase catalyzes the reaction of ADP and Pi to form ATP. Figure 1 presents a model of how the ATP synthase works, called the binding change model. According to this model, the ATP synthase has three subunits that can catalyze the formation of ATP. The catalytic subunits are chemically identical but take on three different conformations, each of which has different binding affinities for the inputs and products of the reaction (ADP and ATP, respectively). The three catalytic subunits of ATP synthase are each in a different one of the three conformations, and they all rotate through the same sequence of these conformations: first loose, which binds ADP, then tight, in which ADP and Pi bind to form ATP, and finally open, from which ATP is released.

The binding change model can be explained verbally, but it is more efficiently communicated with Figure 1. The wedge shapes refer to catalytic subunits of the ATP synthase enzyme complex (in open, loose and tight, conformations respectively). The different wedge shapes only refer to the different binding capacities of the different states; the binding change model does not include claims about the specific structure of the different subunit conformations. Contiguity of the wedge characters refers to co-location of subunits in the enzyme complex; spatial relations are used to convey this general feature of the complex, not the more specific structural claim that the catalytic subunits are contiguous. The horizontal double arrows refer to transitions between different states of the enzyme, in which the same subunit changes to a different conformation—for example, from loose to tight—the diagram doesn't represent the complex as rotating. The linguistic terms in the figure have their usual chemical referents; their position at the ends of hooks represents the addition of those items from the complex during transition from one state to another, and their position at the ends of curved arrows represents the deletion of those items. Positioning terms in the concave part of the subunit symbols means that the item is bound to a subunit in that conformation.

The diagram provides a concise way to express the model, and even more significantly, the diagram uses visible features like contiguity relations to express the key explanatory features of the model. The relations between states of the catalytic subunits, and the coordinated transitions of these states over time, are effectively expressed by spatial relations among parts of the diagram. This formatting aids the production of mental models: comprehension of the binding change model requires generation of a mental model structured in terms of the same key relations which the diagram conveys through spatial relations. For this reason, there is a cognitive advantage to using diagrams rather than verbal expressions of hypotheses. Thus diagrams that express models or hypotheses might provide a very effective way to communicate because they aid in generating the mental model at which reasoning is directed.

Figure 2: Clerk Maxwell’s “physical analogy”, Maxwell 1890.

Visual representations may do more than provide an efficient way to communicate due to facilitating the construction of a mental model. Nersessian (1992) explains Clerk Maxwell’s use of a figure to present an analogy between electromagnetic forces and vortex-idle wheel systems. The analogy involves a mental mapping between key relations from the source domain—a mechanical system—to a target domain—electromagnetism. Perception of the figure involves comprehension of a structure of relations that is shared between the two domains, and by the visible features of the figure. So perception of the figure involves comprehension of this structure. The visual representation thus provides an effective way of communicating the analogy. Nersessian’s discussion suggests that images can do even more. The physical analogy is constrained by the mechanics of the situation: if vortices spinning in the same direction were arrayed in direct contact with each other, they would impede each others’ rotation. This is a feature of the situation that is only implicit in a verbal description of an array of vortices spinning in the same direction. This constraint is met by the inclusion of idle-wheels between the vortices, as shown by Maxwell’s figure (figure 2). Presenting the analogy with a visual representation means using the spatial features of the image to represent features of the source domain: in this case, the rotation of adjacent vortices. This makes explicit higher-order relations like the effect of one spinning element on an adjacent element; you would see, if the array consisted only of vortices spinning in the same direction, that they would impede each other (see Nersessian 1992 figure 3A for her rendition of such an image). The use of a visual representation to convey the analogy, by making the effects of adjacent rotating elements explicit, ensures that the full vortex-idle wheel system serves as the source domain for the analogy, rather than just an array of vortices.

The examples above show how images aid the generation of mental models, and suggest that they can play significant roles in the elaboration of mental models. The visual representations discussed used spatial relations to express conceptual relations important to the model. In these cases, the advantage the images provide is that they make these relations explicit by communicating them through spatial relations. Verbal descriptions of the models can convey the same information, but higher-order relations are left implicit and thus impose a higher cognitive load to comprehend, and increase the risk that critical higher-order relations will be overlooked.

4. Reasoning with mental models and graphs

Graphs also explicitly convey higher order relations: they are designed to represent relations among properties. Reasoning with graphs can include drawing inferences about an even higher-order relation, such as a comparison between two different rates of change of one property with another. Gattis and Holyoak (1996) present a study of the how graphs facilitate such inferences which shows that simply using spatial dimensions to represent properties is not sufficient for effective reasoning with graphs. Gattis and Holyoak(1996) investigate how perceptual operations on graphs are used to draw inferences about the relations the graphs depict. They note various studies in which visual representations not only decrease problem-solving time, but also increase accuracy, which suggests to them that “diagrams highlight the information used in reasoning with internal mental models and provide external support in managing the relations in working memory” (Gattis and Holyoak, 232.) The depictive character of images explains this highlighting: the information is presented via a two-dimensional structure. It is the structure that is used in reasoning with internal mental models. The depiction makes this structure explicit and perceptually available. Graphs offer an additional advantage: they integrate information from two conceptual dimensions. This is a demanding cognitive task, which graphs make easier by representing relations between properties through spatial relations; the conceptual dimensions are integrated through visual perception. Because of the mapping between spatial properties of the image and conceptual properties, the relation between variables can be understood with far less cognitive load.

Gattis and Holyoak show that the cognitive value of graphs requires more than the general feature of using spatial relations to represent relations among properties. The specific way relations between variables are plotted has a significant difference on accuracy of reasoning. It is standard graphing practice to use the horizontal axis of a graph to represent the independent or causal variable, and the vertical axis to represent the dependent variable. This convention—which Gattis and Holyoak call the “slope-mapping constraint”—is a graphing technique which produces a relationship between the slope of a graphed line and the rate of change: a steeper line corresponds to a faster rate of change in dependent variable with change in the independent variable. They assess the cognitive value of the slope-mapping constraint by testing for accuracy of reasoning with graphs formatted with, and without, following this constraint. The authors asked about the rate of change of one variable (the dependent variable) with respect to another (the independent variable), in cases in which the dependent variable is on either the vertical or horizontal axis. Their results show that reasoning is significantly better when the dependent variable is plotted on the vertical axis. This is a format that honors the slope-mapping constraint: faster rates of change in the queried variable with respect to the other property correspond to steeper lines.

Furthermore, they tested graphs in which the variable plotted on the vertical axis preserved the similarity relations commonly exploited in pictures, for example by plotting altitude on the vertical axis in a graph of altitude and temperature. Subjects were shown graphs with a solid line representing a relation between altitude and temperature, and a dotted line (see figure 3.) They were queried about relative rate of change of temperature with altitude represented by the two lines. Graphing altitude on the vertical axis will produce a correlation between slope and the rate of change of temperature with altitude in which a faster rate of change corresponds to a fIlatter line (the opposite effect of the slope-mapping constraint, which produces steeper lines for greater rates of change.) Subjects reasoned more accurately with graphs in which the slope-mapping constraint was honored (temperature was plotted on the vertical axis), than they did with graphs using the pictorial analogy (in which altitude was plotted on the vertical axis.)

Figure 3: The slope-mapping constraint

(From Gattis, M., & Holyoak, K. J. (1996). Mapping conceptual to spatial relations in visual reasoning, Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 231-239. Copyright © 1996 by the American Psychological Association. Reproduced with permission.)

B honors the slope-mapping constraint by plotting temperature, dependent variable, on the vertical axis. As a result, as the rate of change of temperature with change in altitude increases, steepness increases. So the solid line, which is steeper, represents a faster rate of change of temperature with respect to altitude than the relatively flat dotted line. A is a graph that does not follow the slope-mapping constraint: altitude, the dependent variable, is plotted on the horizontal axis. When plotted this way, the dotted line, which represents a slower rate of change in temperature with respect to altitude, is steeper.

These results show that accuracy in reasoning from graphs requires more than just plotting the data so that properties are represented by spatial dimensions. That alone produces a relation between properties that is expressed by spatial relations. This study shows that accuracy improves when the graph is structured so that higher order relations have a certain visible form so that steepness of line reflects rate of change. This suggests that subjects were utilizing an abstract mental model of the higher-order relations, which is structured so that a faster rate of change is modeled as steeper. Formatting graphs to conform to this aspect of the mental model promotes accuracy.

5. Evidence, images, and mental models

Visual representations are frequently presented in support of hypotheses. It is striking that in many cases, the forms of the external representations offered as evidence do not have any obvious similarity to the form of the models they support. Confirmation in such cases cannot be explained by a simple matching of the visible structure of the image to that of a mental model which involves mental imagery that corresponds to the perceived image.

Abrahams et al (1994) present a model of the structure of the ATP synthase complex which supports the binding change model of the synthase mechanism. The diagrams of the models for the ATP synthase structure (for example, figure 4) and the binding change model (figure 1) look completely different. The reasoning involved in taking images like figure 4 as support for the model expressed by figure 1 must be a more complicated matter than generating perceptual expectations, which can then be simply compared on the basis of similarity with the structure of the evidential figure. Figure 4 is a representation of the structure of the C-terminal ends of the proteins making up the subunits of the ATP synthase complex. The overall structure of the complex is sort of like a tangerine, with a central core (?, in blue) and six protein “wedges” arranged around this core. The three catalytic ? subunits, in gold, are distributed evenly around the core. The binding change model says that the three catalytic subunits are always in three different structural conformations. This implies that the complex has an asymmetric spatial structure. The authors cite the visible asymmetry of figure 4 as support for the binding change model.

Figure 4: Ribbon diagram of C-terminal protein chains of the ATP synthase complex. Jan Pieter Abrahams, A.G. Leslie, R. Lutter, and J. Walker (1994), “Structure at 2.8 A resolution of F1-ATPase from bovine heart mitochondria”. Nature, 370 6491:621-8. reprinted with permission from Nature; www.nature.com.

Figure 4 is an example of an image that serves as evidence not because of its particular form, but because it has a more general property. In this image, ribbons represent protein chains, and the spatial features of the image are used to represent spatial features among the protein chains of the complex. The diagram of the binding change model, on the other hand, uses spatial relations to represent co-location and transitions over time: these two diagrams do not relate to each other on one coherent set of dimensions. Figure 4 serves as evidence for the binding change model, because that model doesn’t imply a particular structure, but rather a more general feature. The binding change model abstracts from the structure of the ATP synthase, and from many other details of its mechanism, but it does imply that the complex has a general structure feature: according to the binding change model, the complex is asymmetric. Because the visible features of the model of the complex’ structure are formatted so that their spatial features map onto the spatial features of the complex, the image can be used to evaluate binding change model, by looking for asymmetry in the visual representation. Thus the viability of the model can be evaluated perceptually, to see if the visual representation has that property.

The binding change model is a proposal about the number and functional relations between catalytic subunits of ATP synthase; it is not a complete account of the mechanism by which the complex produces ATP. In particular, researchers suspected that the ATP synthase has a rotary mechanism. This is a very unusual feature in a biological molecule, and though the structure of the ATP synthase is compatible with a rotational mechanism, scientists sought stronger evidence. This enzyme complex is far too small to visualize its movement with a microscope. This problem was solved by attaching a fluorescent filament to the central ? subunit of the complex, which had the effect of amplifying both the size and the signal from the moving part of the complex, and filming the movement of the filament with a microscope. That film shows the filament rotating (available for viewing on the Yoshida and Hisabori lab web site, at the following link: http://www.res.titech.ac.jp/~seibutu/projects/fig/f1rot_1.mov.) Noji et al (1997) publish their findings from the video—which the authors refer to as “direct evidence” for the rotational mechanism—in a paper which presents a series of stills from this film, in which the position of the filament is at different angles in subsequent images.

Both the visual and temporal format of the video are important to how it functions as evidence; it uses the dimensions of space and time to support the rotational mechanism of the complex. These are the dimensions relevant to the claim about the rotational mechanism of the ATP synthase complex, which implies a circular pattern over time. The video of the rotating filament does not, however, look anything like the complex itself would, were it visible (with its core subunit rotating with respect to the outer subunits.) This is another example of an image which serves as evidence for a model not by a simple matching procedure, or due to the specific form of the visual representation. Rather, the feature of the image that provides support is a property abstracted from the particular form of the image (in this case, rotary motion of a line.)

This case also highlights a general feature of images that serve as evidence: they do so in the context of other information than is not presented in the image itself. In this case, information about the experimental situation is critical to the evidential role of the video: in particular, the causal link between the activity of the enzyme, which isn’t visible in the video, and the movement of the filament, which is visible. So the function of this moving image as evidence depends on more than what is visible in the image, or even its content. This is a general feature of visual representations presented as evidence: their warrant as evidence derives not just from their content, but from background knowledge that supports their reliability and relevance to the hypothesis in question.

6. Conclusion

These examples show several ways in which images might be involved in scientific reasoning, if that reasoning takes the form of constructing, changing, drawing inferences from or supporting mental models. The results show that it is not merely matching the structure of the mental model that is important to the use of scientific images. Structure-matching could decrease the work needed to generate a mental model, but the value of explicit presentation of higher-order relations goes beyond cognitive efficiency. In addition, other cognitive functions, like evidential reasoning with images, are often not a matter of first-order matching between image and model structure. Rather, scientists abstract features from the evidential images which provide the relevant support.

The importance of more abstract features of scientific images is also shown by the result’s Gattis and Holyoak present from their study of reasoning with graphs. This shows that humans reason in terms of abstract mental models which can have structures which impose constraints on accurate reasoning with images. Specifically, they showed that drawing inferences about higher-order relations (in this case, making a comparison between different rates of change represented by a graph) depended on a specific visual formatting which conforms to the structure of the mental model.

The tactic of studying scientific reasoning with images by thinking about how they would relate to cognition involving mental models has led to a result that might be surprising, at least in light of everyday assumptions about pictures and their informational density. From the perspective of mental models, it seems that scientific images are valuable not because—relative to text—they present rich perceptual content, or convey more detailed information, but rather due to their more abstract features.

References

Abrahams, Jan Pieter, A.G. Leslie, R. Lutter, and J. Walker (1994), “Structure at 2.8 A resolution of F1-ATPase from bovine heart mitochondria”. Nature, 370 6491:621-8.

Cooper, L.A. and R. Shepard 1973. “Chronometric Studies of the Rotation of Mental Images” in W.G. Chase (ed.), Visual Information Processing. New York: Academic Press.

Maxwell, James Clerk 1890. The Scientific Papers of J.C. Maxwell, ed. W.D. Niven. Cambridge: Cambridge University Press.

Gattis, Merideth and Keith J. Holyoak 1996. “Mapping Conceptual to Spatial Relations in Visual Reasoning”, Journal of Experimental Psychology: Learning, Memory, and Cognition 22:1 231-239.

Nersessian, Nancy (1992), “How do scientists think? Capturing the dynamics of conceptual change in science”, in Ronald Giere (ed.), Cognitive models of science. Minnesota Studies in the Philosophy of Science, v. 15 . Minneapolis: University of Minnesota Press.

Noji, H., R. Yasuda., and K. J. Kinosita 1997. “Direct observation of the rotation of F1ATPase”, Nature 386: 299-302.

Yoshida and Hisabori lab web site: http://www.res.titech.ac.jp/~seibutu/home.html