1:  What is enaction¹ in the context of music and how does it become enfolded in current notions of performance?

Implicit in the experienced musician's understanding of the relationship between action and sound, between performance gesture and musical phrase, is a more or less complex internal representation of the dynamics of an instrument.  This internal representation, built up over many years of practice, provides the skilled performer with an understanding of the instrument that is free from the context of any one piece. Such contextual independence provides the means for continually fresh interpretation and flexibility in performance².

In considering how concepts of enaction relate to performance, to the performer-instrument interaction loop, one question which must be considered is how is the musician’s working model of the dynamics of a system as complex as a musical instrument built up?  Moreover, how can enaction, in Bruner’s broader definition of ‘Knowing by Doing’ (Bruner, 1968) account for the musician’s ability to predict the outcome of actions which they may never have had to produce before – a nuance in performance that is produced ad libitum.  While we cannot answer these questions definitively, the following discussion will, we hope, raise some of the issues we consider to be keys to approaching such questions.

The concept of body-mediated or embodied interaction, of the coupling of interface and actor, has become increasingly relevant within the domain of HCI.  With the reduced size and cost of a wide variety of sensor technologies and the ease with which they can be wirelessly deployed, on the body, in devices we carry with us and in the environment, comes the opportunity to use a wide range of human motion as an integral part of the interaction with all sorts of applications.  The term ‘Embodiment’, defined by Dourish as ‘the transition from the realm of ideas to the realm of everyday experience’ (Dourish, 2000), encompasses not only physical embodiment (of objects such as tables and chairs), but also embodied actions such as speech and gesture.  For Dourish, the notion of embodiment is related to Heidegger's phenomenological approach to the world and the way we act upon it.  Heidegger distinguished between two categories of interaction -those where the world is present ("vorhanden") and where the world is acted upon ("zuhanden"). Dourish translates these concepts as "present-at-hand" and "ready-to-hand", and suggests that embodiment is equivalent to Heidegger’s concept of “zuhanden”.  Dourish argues that a Human-computer interface is, when acted upon, "ready-to-hand".

How does the notion of embodiment relate to enaction?  Varela, Thompson and Rosch define the relationship thus:

 “By the term embodied we mean to highlight two points: first, that cognition depends upon the kinds of experience that come from having a body with various sensorimotor capacities, and second, that these individual sensorimotor capacities are themselves embedded in a more encompassing biological, psychological, and cultural context.  By using the term action we mean to emphasize once again that sensory and motor processes, perception and action, are fundamentally inseparable in live cognition. … the enactive approach consists of two points: (1) perception consists in perceptually guided action and (2) cognitive structures emerge from the recurrent sensorimotor patterns that enable action to be perceptually guided.”  (Varela et al, 1991)

What concerns us here, then, is the consideration of musical performance as a series of perceptually guided actions embedded in a specific biological, psychological and cultural context, the realisation of a piece of music.

2:  Enaction and Playability

One challenge in applying the concept of enaction to musical performance is in how to account for actions that occur in a time frame that does not allow for a simple definition of ‘perceptually guided action.’  Highly skilled musicians playing, for example, very rapid sequences of notes, do not have time to account for each event as a distinct perceptual unit – they rather perceive a series of actions, often governed by a shared expressive goal, as a musical phrase or sub-phrase.  From this arises the notion of an ‘instrumental  Gesture’ (Cadoz and Wanderley, 2000)  and the associated notion of ‘gestural control’, notions which are related fundamentally through the actions by which the music is brought into being in performance.

It is no accident, then, that theoretical discussions of the design of new electronic musical instruments usually focus on the theme of gesture and the technology to capture and retain some aspect of the expressive properties gestures convey.  While most such discussions take as their starting point the notion of performance as defined by the existence of a musical score or a thematic seed for an improvisation, Iazzetta (2000) and Fells (2004) have drawn attention to certain elements of the concept of embodied action as a means of encapsulating the requirements of digital musical instrument interfaces.  For Iazzetta, the body serves as the anchor for gestures. His notion of the body is hence an instrument for gestures. However, he suggests that the action of the body is not contingent on cognition, a view that is closer to that of Peirce's structuralist semiotics than Valera’s notion of embodiment. Fels, on the other hand, describes the notion of control but argues that the mechanics of control is insufficient. He claims that a desirable musical instrument is based on the notion of intimacy in the relationship between performer and instrument. While there is still an implied connection between the actor and the environment of the instrument, it is described not in terms of embodied action, but in terms of intimacy suggesting a somewhat more emotive and less physical relationship.  Between these two approaches, it seems, is a third which builds outwards from the gesture as embodied action, as the movement required to produce music, toward a theoretical approach that can unite the notion of the instrumental gesture with a framework for development of digital musical instrument interfaces with embodied action at its core.  For the present authors, it has been important to try to map the space between the literature on the instrumental gesture on the one hand and that on gestural control on the other and we present our summary of these two areas in the sections below.

2.1:  The Instrumental Gesture

In their seminal article, Cadoz and Wanderley (2000) define the ‘instrumental gesture’ as the manipulation of the instrument necessary to mechanically produce the required sound.  Thus, the instrumental gesture implies a knowledge of the possibilities for action afforded by the instrument.  Above and beyond this knowledge, though, the instrumental gestural channel also encompasses communicative as well as interactive possibilities and supports both the transmission of information to the instrument and reception of information from the instrument.  Of particular interest to the present authors is the fact that, in cases where a mechanical coupling between player and instrument exists, such a two-way flow of information is mediated in part by the haptic channel.

In their definition of the instrumental gesture, Cadoz and Wanderley further identify three interdependent functions:

• material action, modification and transformation of the environment — the ergotic function; 

• perception of the environment — the epistemic function;

• communication of information towards the environment — the semiotic function.

How do these functional properties of the instrumental gesture relate to notions of enaction?

With regard to the first, ergotic, Cadoz and Wanderley suggest that there is no communication of information between the hand and the object, only energy communication.  Forces applied to the object cause deformation and displacement and part of the applied energy is fed back to the gestural channel.

“The ergotic function is the one that allows the differentiation between communicative and interactive goals of the gesture.” (Cadoz and Wanderley, 2000.)

The second function, the ‘epistemic function’, relates to our perception of the environment of the instrumental gesture.  While Cadoz and Wanderley suggest this is primarily mediated by our combined tactile and kinesthetic senses, present authors would argue that our visual and auditory senses also play a part in monitoring the success or otherwise of a given gesture.  Thus, our knowledge of the potential of an action might rest on a richly multisensory understanding of the behaviour of an instrument.

The third function identified by Cadoz and Wanderley, the semiotic function, is that of meaning, of communicative intent, and as such is closest to traditional notions of gesture such as gesticulation, pantomime, etc.  For music, it is the function that acts to invest a series of actions that produce notes with a cohesive interpretive trajectory which correlates parameters of timing and amplitude (musical dynamics), to ‘express’ a performer’s interpretation of a piece.  Interestingly, Schaffer notes that it is possible for such interpretive gestures to subsume previously learned sequences of movements (phrases, etc.) to ‘re-interpret’ known passages of music on the fly. For this reason, music serves as a compelling example that challenges hierarchical descriptions of skill acquisition and motor control (Shaffer, 1993.)

To summarise, the notion of the ‘instrumental gesture’ encompasses distinct but interdependent functional elements, namely transmission of energy to produce sound, perception of the state of the system and of possibilities for affecting this state, and communication, through action, of musical intent. 

2.2 Gestural Control

Loosley defined, gestural control addresses the question of how to map the capabilities of the human sensorimotor system to the parameter space of the instrument being played, in order to provide the performer with maximum control of music's four fundamental elements - time, pitch, timbre and amplitude. For the performer, manipulation of these dimensions is embodied in physical actions such as striking a piano key or bowing a string.  As noted above, in real instruments these ``instrumental gestures'' are determined by the sound-producing mechanisms of the instrument.  The sound produced, in turn, carries the characteristics of the movement that gave rise to it (Cadoz, 1988.)

A by-product of sound synthesis techniques such as physical modeling (Smith, 1998) is the ability to decouple the synthesis of an instrument's sound from the physics of the instrument's sound-producing mechanism.  Thus the affordances of a synthetic music controller can be very different from those of the instrument being controlled.  In other words, there is no longer a direct mapping between the instrument and the instrumental gesture.  A percussive controller such as a piano keyboard, for example, might be used to control a physical model of a bowed string.  The question then arises:  What are the implications for the design of computer-based musical instruments and their supporting hardware and software protocols of decoupling the instrumental gesture from the sound-producing mechanism? 

As evidenced from the work of the Enactive Interfaces project itself, one potentially powerful approach is to turn to embodied action as the anchor for a theoretical framework that could draw upon the instrumental gesture as the basis of new methodologies for interface and protocol design.  For the present authors, this starting point has lead to the realisation that, for certain classes of sounds, there exists physical phenomena that have equally powerful tangible associations, the binding factor being the underlying physics of the phenomena being experienced.  But, more importantly, it would appear that this binding need not be absolute – as long as the tangible and auditory experience are tightly coupled in time and share common affordances or opportunities for action.  Thus if an interface that offers the opportunity to be banged is coupled with the sound of a wrecking ball hitting the side of a building, an action which is impossible in the real world – the demolition of a building with your bare hands – can be very compellingly supported.  In the following section we will describe the realisation of one such example, PebbleBox, an interface for the control of granular synthesis models. 

In summary, the notion of gestural control, in the context of enaction, offers the opportunity to expand the concept of the Instrumental Gesture.  By taking as a starting point a physical system that produces sound in response to action and finding other physical systems with closely related behaviours, it is possible to define strategies for mapping actions to whole classes of sounds that have common physical roots.  The crucial point is that the classes of sounds defined in this way will most likely already share certain affordances or opportunities for manipulation.

3:  Building Enactive Instruments:  Some Initial Examples

The overarching goal of our work on haptic controllers for computer-based musical instruments in the context of the Enactive Interfaces project is to uncover instances of coupling, however loose, between the haptic and auditory senses and to build on these couplings to develop new paradigms for instrument control. The examples presented here represent a sub-set of such controllers, those based on interactions that are mediated by physical objects, the properties they embody and the manipulation strategies they invoke³. 

At the heart of our design of haptic controllers for granulated sounds is recognition that there exist a class of sounds which are produced by our actions on objects in the world.

Thus dragging, dropping, scraping and crushing give rise to correlated touch and sound events (Rocchesso et al 2003.) As noted earlier, such events also bear many signatures of other physical characteristics of the materials and actions involved.  However, it is possible to imagine a further class of events where the feel of an object and the sound it produces are less strongly correlated - for example, when playing with pebbles in ones hand, the haptic sensation one feels is that of the pebbles against the hand, while the sound of the interaction stems from the colliding of pebbles within the hand. This loose correlation between feel and sound is appropriate for this experience and in its looseness provides an opportunity to decouple the haptic experience from the sound source. This is the opportunity we build on in our granular synthesis controllers.

Since the current goal was to build a controller that couples the feel and sound of granular events, it was important to incorporate into the interface the manipulation of elements that could objectively or subjectively give rise to granular sounds. Two different interaction paradigms were developed, playing with a hand-full of pebbles (PebbleBox) and crushing a bag of brittle material (CrumbleBag)⁴. Both are somewhat complex environmental events, whose temporal patterns give rise to important perceptual cues (Keller and Berger, 2001; Warren and Verbrugge, 1988)

Therefore, there is a need to sense these temporal events. This poses a number of problems. Firstly, given the nature of the sounds of interest, the events are likely to be spatially distributed. Moreover, the sound-producing mechanism may be internal to the objects interacted with crinkling paper, or may be a result of their destruction (for example crushing cornflakes.)

Finally, while the coupling between temporal events as they are perceived by both the haptic and auditory system should be relatively tight, we are interested in leaving other parameters such as dynamics and timbre open for exploration by the performer.

 Figure 1. Pebblebox

The PebbleBox is designed to allow for direct manipulation and ecological behaviours of objects in a relatively unconstrained way. See Figure 1. The design consists of a foam-padded table with an inlaid actively powered microphone. The purpose of the foam is to eliminate the possibility of objects colliding with the container and to damp the sounds of objects dropped or rolled inside the box. However, interactions and disturbances are still picked up by the embedded microphone. Additionally, the microphone picks up interactions in a limited range above the device, i.e. the interaction of objects held in the hands just above the box.

Typical sounds are the collision of objects with the foam padding and collisions between objects. Haptic feedback is a result of the direct manipulation of the objects in the PebbleBox.

4: Summary and Conclusions

In this paper, we have sought to take the concept of enaction as the starting point for a discussion of alternative strategies for the design of interfaces to computer-based musical instruments and strategies to map the affordances of such interfaces to parameters of the sounds they control. Taking as a starting point the considerable bodies of literature on the Instrumental Gesture and on Gestural Control, we have sought to find a way to relate their shared roots in embodied action as a means for generation of sound to suggest that the grounding of the instrumental gesture in the physics of sound production might become a point of departure for new classes of computer-based musical instruments.  These instruments might depend, we suggest, on finding common properties of related physical systems that share certain affordances, opportunities for action that give rise to both auditory and tangible responses.  Citing our recent work on PebbleBox as an example (O’Modhrain and Essl 2004), we suggest that this approach can enable instrument builders to define strategies for mapping actions to whole classes of sounds that have common physical roots.  The crucial point is that the classes of sounds defined in this way will most likely already share certain affordances or opportunities for manipulation, a property which will help to determine the kinds of instrumental gestures it will support, reintroducing for computer-based musical instruments the notion of a dependence of both the actions on the instrument and the sounds produced by the instrument on shared physical laws.

Footnotes

1. Elena Pasquinelli shared an unpublished draft of a wonderfully comprehensive review of enaction in philosophy and cognition with us during the preparation for this draft (Pasquinelli, 2004). We gratefully acknowledge the helpful impact of this manuscript on our writing and thinking.

2. The term ‘representation’ is used here in the loosest sense to denote some internal memory that is the basis for the expectation that a given action on a musical instrument will produce a given sound output.

3. For more details on experimental investigations into the importance of haptic feedback for musical performance see Gillespie (1996), Roven and Hayward (2000), O’Modhrain (2000) and Nichols (2003).

4. For a more detailed discussion see O’Modhrain and Essl (2004).

References

Bruner, J. (1968). Processes of cognitive growth: Infancy.  Worcester, MA: Clark University Press.

Cadoz, C. (1988) “Instrumental Gesture and Musical Composition,” in Proceedings of the International Computer Music Conference (ICMC-88), Cologne.

Cadoz, C. and Wanderley, M. (2000) “Gesture – Music” in Trends in Gestural Control of Music, Battier, M. and Wanderley, M., eds. Paris: Editions IRCAM, 2000, pp. 71-93.

Dourish, P. (2000) Where the Action Is, Cambridge, MA: MIT Press.

Fels, S. (2004) "Designing for Intimacy: Creating New Interfaces for Musical Expression,” Proceedings of the IEEE 92:4, 672-685.

Gillespie, R. B. (1996) “Haptic Displays of Systems with Changing KinematicConstraints: The Virtual Piano Action,” in Mechanical Enginnering. Stanford, CA: Stanford University, 1996.

Iazzetta, F. “Meaning in Musical Gesture.” in Trends in Gestural Control of Music, Battier, M. and Wanderley, M., eds. Paris: Editions IRCAM,, 2000, pp. 259-268.

Keller, D. and Berger, J. (2001) “Everyday sounds: synthesis parameters and perceptual correlates.” In Proceedings of the VIII Brazilian Symposium of Computer Music, Fortaleza, Brazila.

Nichols, C: (2003) “The vBow: An expressive Musical Controller Haptic Human-Computer Interface,” in Center for Computer Research in Music and Acoustics (CCRMA). Stanford, CA: Stanford University, 2003.

O'Modhrain, S. “Playing by Feel: Incorporating Haptic Feedback into Computer-Based

Musical Instruments”. PhD thesis, Stanford University, Palo Alto, California.

O'Modhrain, S. and Essl, G. (2004), “PebbleBox and CrumbleBag: Tactile Interfaces for Granular Synthesis,” in the Proceedings of the Conference for New Interfaces for Musical Expression, Hamamatsu, Japan, pp. 74-79.

Pasquinelli, E. (2004) “Enaction. A State of the Art,” unpublished manuscript.

Rocchesso, D. and Fontana, F. eds. 2003 “The Sounding Object.” PHASAR Srl, Florence, Italy, 2003.

Available online at http://www.soundobject.org/SObBook/

SObBook_JUL03.pdf.

Roven, J.  and Hayward, V. (2000) "Typology of Tactile Sounds and their Synthesis in Gesture-Driven Computer Music Performance," in Trends in Gestural Control of Music, Battier, M. and Wanderley, M., eds. Paris: Editions IRCAM, pp. 355-368.

Shaffer, L. H. (1993) "Motor Programs and Musical Performance," in Attention: Selection, awareness, and control: A tribute to Donald Broadbent, A. Badley and L. Wisekranz, Eds. Oxford, UK: Clarindon Press, pp. 135-151.
 

Smith, J. O. (1998) "Principles of Digital Wave-guide Models of Musical Instruments," in Applications of Digital Signal Processing to Audio and Acoustics,, M. Kahrs and K. Brandenburg, Eds. Boston/Dordrecht/London: Kluwer Academic Publishers, pp. 417-466.

Varela, F., Thompson, E., Rosch, E. (1991). The Embodied Mind. Cambridge, MA:  MIT Press.

Warren, W. H. and Verbrugge, R. R. (1984) “Auditory Perception of Breaking and Bouncing Events: A Case Study in Ecological Acoustics.” Journal of Experimental Psychology, 10:5, pp. 704-712.