The Danced Sound Sculpture

Julia H. Schröder

When the dancer Tomoko Mio danced with Jutta Ravenna’s Data Window sound sculpture SesamSesam (2017), her gestures influenced the sound or indeed the music that the sculpture emitted. In the following, such interactive “danced sound sculptures” are analyzed. A sound sculpture is an art object which emits sound, as the one by Jutta Ravenna or the sounding sculpture-relief Rythme d’ombre (rhythm of shadow) by Peter Vogel. They are both interactive: if a shadow from a visitor falls on one of the light sensors in Vogel’s relief, a new soundtrack starts. Thus, the dancer becomes the interpreter of the music evoked by her gestures.

Peter Vogel’s Sound Sculptures

While experiencing Peter Vogel’s sounding sculpture-relief Rythme d’ombre (2004) in a gallery (figure 1),¹ I was struck by the way the art work made me move: when my hand passed a sensor, a sound started, beginning with a fast bass drum pulse (the bassline). Wanting to explore more sounds, I moved on. One after the other, I triggered some percussion rhythms, then an electronic, melodic pattern and finally a kind of looped bongo-pattern. They were triggered directly, as a one-to-one mapping of gesture to sound. But the individual soundtracks played for a while before fading out—until being triggered again by walking back. The result was a composition of different rhythm tracks, patterns, and recurring short melodic fragments, like an electronic dance music piece.

And indeed, Peter Vogel called a similar sound sculpture, created for the Sonambiente sound art exhibition in Berlin in 1996, a “techno sound wall,” and gave it the title Rhythmic Sounds. It has eighteen tracks, i.e. eighteen sensors, and is almost six meters long.²

On Vogel’s website, a video of the 1996 dance performance by Bridge Markland and Rhythmic Sounds exists.³ One can see the relief mounted at approximately chest height on a wall, with the dancer moving in front of it (see figure 2). The lighting makes her shadow fall on the relief. When she moves past one of the eighteen light sensors, her shadow starts a new sound track. There is a build-up in both dance and music; she moves progressively faster and the music gets denser accordingly. In Markland’s techno interpretation, one starts to lose the notion that she triggers the sounds. Perfectly synchronized dance and music meld together, and her repetitive gestures equal the looped soundtracks: Dancing the music is entrainment.

Figure 1: Peter Vogel, Rythme d’ombre, Mixed Media (Sound), ca. 400 cm x 16 cm (2004), photographed at DAM Gallery Berlin 2018. © Julia H. Schröder

Figure 2a and b: Bridge Markland dancing with Peter Vogel’s Rhythmic Sounds “techno sound wall”-Sculpture at E-Werk Freiburg in 1996; video still from http://www.petervogel-objekte.com/PerfTanz.html. © Peter Vogel’s estate and Bridge Markland

While the artist classifies the “Klangwände” (sound walls) as ‘installations’ and formerly in an upright form “Klangstelen” as ‘environments’,⁴ I would categorize them as ‘sound sculptures’. They are clearly confined objects, and would thus be classified as sculptures rather than installations. The fact that they need lighting and emit sound into the room could be understood as installation-like, as well as the fact that there are loudspeakers attached, which might stand in another corner of the room. Usually, the loudspeakers are not integrated into the relief.⁵ But the acoustic extension into the room holds true for any sound sculpture, whereas a sound art installation is commonly understood as an artistically created space or extended outdoor art work. A sound art installation cannot be reduced to an object and is often ephemeral.

The visible structure of Vogel’s “Klangwände” lies in their components, which are electronic circuitry elements like transistors, oscillators, capacitors, etc.⁶ They form an intricate structure, with a certain resemblance to a five-line staff in musical notation. Vogel spoke of “materialized scores”⁷ to stress the fact that his sculptures were not sounding by themselves, and only emitted sound when ‘played’ by a visitor or a dancer.

During his student days, Peter Vogel danced, choreographed, and composed electroacoustic music while studying to be a physicist. Combining his interests in his artistic work, he started in the late seventies with interactive “environments”⁸ which were also danced in. In 1975, Vogel installed three of his upright interactive sound sculptures with four photocells at the contemporary music festival Donaueschinger Musiktage; these were activated by the visitors. Two years later, the same three sculptures were first danced with at the Music Biennale in Zagreb. From his experience with dancers in his environments, he developed the first horizontal wall-mounted sound sculpture for dance in 1979. Later he adjusted the concept to current musical genres, such as minimal music⁹ and electronic dance music. In fact, the dance and the musical style are markedly diverse, being updated with new styles in new periods.

Vogel writes that he usually works with individual dancers for a while.¹⁰ They improvise with possible musical structures and the dancers get to choose their favorite sounds, which they then start to relate to certain dance gestures. At this point, Vogel selects the definite sounds and the dancer determines the dramaturgy of the macro form, whereas the meso and micro form will be improvised during the performance. The dancers should react flexibly to the sounds.

Figure 3a and b: Modern dancer Christine Brodbeck dancing with Peter Vogel’s first Große Klangwand sculpture in Zurich in 1979; video stills from http://www.petervogel-objekte.com/PerfTanz.html. © Peter Vogel’s estate and Christine Brodbeck

In the 1979 sound sculpture, there were thirteen sensors with pitched sounds,¹¹ some of them single tones which either reacted immediately or only after being shadowed for a while. Other sensors started melodic fragments. Vogel calls the tone repertoire “atonal.” This first sound wall was danced with by the modern dancer Christine Brodbeck.¹² Whereas Bridge Markland’s dance for the Techno Sound Wall was based on repeated, fast paced rhythmic loops—a list of the musical patterns is published,¹³ but could equally be transcribed from the video documentations or the sound sculpture itself—and included dancing along the row of sensors, Brodbeck chose different areas of the sculpture to explore their different sound possibilities (figure 3a). During her dance, she repeatedly moved in a low position (figure 3b) as to not throw any shadows on the sensors until she had reached another area and thus other sounds (e.g., a silent floor episode [1 min. in the video excerpt], or moving to the high frequency at the end of the video excerpt [3:30 min.]).

A third dance style with this type of sound sculpture, Japanese Butoh, was performed by Mitsuyo Uesugi in 1994.¹⁴ In the beginning we hear mostly percussive sounds and Uesugi dances in an almost robotic way—very slowly exploring with one arm, adding sideways torso movements, imitating the melodic pattern. After establishing a complex but minimalistic sounding rhythmic pattern, she moves away from the sculpture (1:30), listening to the music and continuing on to face the other end of the relief. At 1:40 she moves towards the right end of the sculpture, now facing it, which adds a different melodic pattern resulting in an almost chaotic musical moment, until the first pattern fades out and the new one takes over. She dances to it with gracefully undulating arms (figure 4a) on the right side of the sculpture, demonstrating how the music fades away when she moves into the room (2:30). In the last part she triggers the same melodic pattern again, but in a hunched position with high shoulders and fists (figure 4b), as well as strained and repetitious movements, changing her interpretation of the same music through different dance expression.

Figure 4a and b: Butoh dancer Mitsuyo Uesugi dancing with Peter Vogel’s Sound Wall sculpture at gallery TOM in Tokyo in 1994; video still from http://www.petervogel-objekte.com/PerfTanz.html. © Peter Vogel’s estate and Mitsuyo Uesugi

In Vogel’s interactive sculptures for dance, the adaptability to different dance and musical styles was unexpected. Although a certain position in relation to the sculpture is needed to trigger a specific sensor at a specific time, the triggering gesture is irrelevant, as is the body part: it can be triggered by the shadow of the hand, a raised foot, a passing head or torso. The dancers are free in their stylistic choices of movements.

Vogel’s kinetic and sound sculptures are primarily exhibited in galleries and museums. It is interesting, therefore, to note that the artist speaks of the recipient as spectator/performer/dancer “(Betrachter/Spieler/Tänzer),”¹⁵ thus ascribing a triple role to the visitors who become equally instrumentalist and dancer when interacting with this sound sculpture.

The spectator/performer/dancer can influence the combination of the sound events, as well as, in some cases, their duration and dynamics. He or she must move to produce the triggering shadows. The visitor is almost provoked to dance to the music, which is triggered and thereby arranged in time.

Interactivity: Dancer = Musician?

In this article, I am interested in the interactivity between a dancer and a sounding sculpture: if the dancer can influence the music, does she act as musician?

She does, to a limited degree. If she has agency over the sonic result, she assumes—at least partially—the role of a musician. Dancing with Vogel’s sculptures, she selects the sound she wants to hear, the order and timings of the sounds, and the density of the sound layers. The individual soundtracks are already composed and limited in number. She cannot play a melody on this sound sculpture, as she could on a musical instrument. As Vogel writes, the difference between his sound objects and a musical instrument is that instruments faithfully reproduce the same sound result when played, whereas his sensors can play differing sounds when triggered repeatedly, if so designed.¹⁶ In musical terms, there is an open form or mobile form with a reservoir of sound structures which will be ordered differently with each performance. Vogel names non-linearity as differentiation between his sound walls and a musical instrument. But the most important difference is that the sound sculptures play composed sequences (i.e. already compositionally formed material), whereas an instrument plays separate tones.

An example of such a musical instrument is Lev Termen’s (Leon Theremin’s) Terpsitone from the 1930s, which was a development of his earlier electronic musical instrument, the Theremin or Theremin Vox (1921).¹⁷ With the help of two antennae, musical tones can be influenced in pitch and loudness without touch. Whereas the Theremin is played with hands in the air, the Terpsitone is played with the whole body. The performer seems to be dancing while eliciting the music.¹⁸ Andrey Smirnov writes:

There is an insulated metal plate beneath the surface of the platform. As the dancer bends towards it, the electrical capacity is increased, and thereby the pitch of an oscillating tube circuit is lowered; as the dancer rises on tiptoe, for instance, the pitch of the oscillator is increased. Thus the motions of the dancer are converted into tones that vary in exact synchronism with his or her pose. In fact, the motion of either an arm or a leg is sufficient to produce a noticeable change of tone.¹⁹

The Terpsitone was first performed on by the violinist and Theremin-virtuoso Clara Rockmore at Carnegie Hall in New York in 1932.²⁰ At the same recital several inventions were performed upon, the Terpsitone standing apart by its focus on body movements. While playing the musical instrument, the musician was seemingly dancing. Lev Termen worked with several dancers;²¹ however, to the best of my knowledge, this collaboration did not result in performance series.

Although the performer is ‘dancing’ whilst playing, the construction of the Terpsitone makes it very difficult for a dancer to perform upon: a glissando can be played with a long movement, but playing a melody such as the Theremin-virtuoso Clara Rockmore did requires more musical training.

Summarizing, I would call the Terpsitone a musical instrument which aims at a full body performance, not a sound sculpture.

We find similar systems of non-touch sound-control,²² or ‘hands-free natural user interface’ frequently today. The systems are often video-based, monitoring the camera-field and acting if motion occurs in specific parts of that field, as in Jutta Ravenna’s SesamSesam. Another example is David Rokeby’s Very nervous system (1986-90).²³ Other interfaces use different sensors, e.g. light sensors in Peter Vogel’s sound sculptures. Technically, they belong to the research field of human-computer-interaction (HCI). Interfaces between the machine and the human user govern new musical instrument design. Well-known examples include composer-performer Laetitia Sonami’s gestural controller “Lady’s Glove,” and Michel Waisvizs’ “The Hands”. The community exchanges their inventions at The International Conference on New Interfaces for Musical Expression (NIME).

Equally, we find gestural controllers in video games. Musical computer games can be divided into ‘rhythm action games,’²⁴ like Dance Dance Revolution (1999) in which one steps on given points of a platform (the controller), or Guitar Hero (2005) in which the gamer presses keys on a guitar-shaped controller synchronous to the playback of a song. The gamer in these examples synchronizes his movements to a given song.

’Electronic instrument games,’²⁵ on the other hand, do not necessarily use a special controller, but they offer the possibility to influence the musical outcome, either in the form of improvisation or even composition of some elements; they are also called ‘sound toys.’ Since Andrew Dolphin defines these ‘sound toys’ as interactive (“sonic-centric systems in which the end user may trigger, generate, modify, or transform sound”²⁶) I would call them ‘music toys.’ They often allow for improvisation in a specific musical style. In this respect, ‘music toys’ resemble Peter Vogel’s sound sculptures.

Looking at dance games or ‘rhythm action games,’ we could compare their controllers to the Terpsitone. Both are new forms of interfaces to control an instrument, i.e. the computer game or the electronic music instrument.

In experimental music these types of interfaces of musical control are even more diverse, ranging from data gloves²⁷ to fifteen meter long wires connected to an internet outlet.²⁸ In dance performances, various sensors can be used to influence the music in the performance, including accelerometers which measure the movement of a dancer in space.²⁹

If the dancer can control many sound parameters with her movements in real-time, does she become the composer of the resulting live-electronic music?

Sonification vs. Composer-Dancer

The technical possibilities can be viewed in two ways: theoretically, we can sonify certain features of a dance easily. Sonification means an automatic translation into an acoustic signal. A controller can translate an arm rotation into a rotating sound—or any other sound type specified. If we understand the dancer as having musical agency on the other hand, we would consider her to be the composer who selects and arranges the musical material through her dancing. There are certainly steps in-between, such as music that is automatically synchronized to a dancer’s gestures, or the notion of the dancer as a musician who interprets a composition. If we see an automatic translation of dance into sound or understand the dancer as composer, I would ask: is the result music? If I consider the dancer as musician, I would ask whether she achieved a musically satisfying interpretation (figure 5).

My last example is about limited musical improvisation by dance with a sound sculpture as well. It employs non-touch sound control and a variable mapping of sounds to gestures, which allows for a modification of both in real-time. In this case, the metaphor of the dancer as a conductor has been used.³⁰

Figure 5: Graphic depiction of functional roles and resulting analytical questions in danced sound sculptures. © Julia H. Schröder

Jutta Ravenna SesamSesam

Jutta Ravenna’s SesamSesam (2017) is an interactive sound sculpture, which can be performed with via bodily motion and influenced via smartphone or other mobile devices with internet access. It was installed 2017 at a festival on sonification, curated by Jutta Ravenna herself, and reflects her double interest in sonification of mobile data and bodily motion.³¹

Figure 6: Jutta Ravenna’s sketch for SesamSesam (2017): Sensor space with dancer in front of the interactive sound sculpture Data Soundwindow. © Jutta Ravenna

In front of the sculpture, a conical space of approximately six meters is scanned from inside the sculpture by a camera—actually a natural user interface called Kinect, developed for video games as well as medical research and surveillance. The dancer’s positions are analyzed in a real-time process and allow for a more complex mapping of sound parameters on gestures. It is a three-dimensional sensor, meaning the analysis program can distinguish human movements such as turning and arm gestures, and it maps them to specific sound events, composed by Ravenna. Ravenna pre-defined a movement vocabulary that was mapped onto a sound archive of composed musical material. There she specified certain arm movements, but also head-rolling and so on. That means there is, for example, an entire repertoire of arm movement for the dancer to learn, including which movements control certain sound events; the dancer’s position in space governs parameters such as the dynamics, tempo, and duration of the sound event triggered. Could a random dance result in an interesting musical result? Possibly, but not necessarily. In that sense the sound sculpture would sonify the dance–simply making it audible. Depending on the dancer, her style, and the specific sounds the sculpture was equipped with, different results would be achieved.

In the published sketch (figure 6) Ravenna visualizes the space in front of the Sound Sculpture, the data window, as partitioned into seven zones of distance from the sculpture and two sides, thus fourteen different fields. If a body-part enters one or several of these fields, it is analyzed and triggers specific musical structures or elements. As a video documentation of one of the three dance performances, Dancing Sound—Conducting Data (2017) by Tomoko Mio shows (figure 7), the back area of the room at the beginning and the end of the excerpt were eliciting percussive sounds. When the dancer moved to the front, one could hear pitched sound events. This mapping of specific sounds to areas can change and there are states with extended sound textures added.

Tomoko Mio danced with Jutta Ravenna’s sculpture several times over the course of the three-day festival in the foyer of the Villa Elisabeth in Berlin in October 2017. This way, she was able to show a range of possible interactions with the sounding sculpture SesamSesam. The dancer and the sound artist had worked together for a period before its final installation on-site. In this development of the movements and the sound mapping, they laid the foundation for the performance Dancing Sound—Conducting Data.

SesamSesam is part of a series of sound sculptures, including Sound Windows, which Ravenna started in 1995.³² They are assembled from recycled computer circuit boards and use their specific sounds, mostly consisting of beeping and clicking and whirring. Ravenna called this “reanimation,” as in the sculptures consist of “reanimated computer circuit boards.”³³ They are sonically reacting to visitors, for example to their movement and distance, as measured by ultrasound sensors. Another realization is a sounding column of these circuit boards, in this instance installed in the foyer of a German radio building from 1930 where it formed the title-giving “ninth column.”³⁴ Apparently, Ravenna included light and humidity sensors as well, thus changing the art works of this series constantly.

Figure 7: Tomoko Mio dancing with Jutta Ravenna’s sound sculpture SesamSesam (2017) in the foyer of Villa Elisabeth, Berlin (October 2017). © Arata Mori

This sculptural aspect is also present in the SesamSesam (2017) installation: when I asked Jutta Ravenna whether a passing dog would make music with her sculpture, she answered that firstly, the sensorial system recognizes human beings and is even used in border surveillance between South and North Korea—thus stating the agency of the human dancer—and secondly, SesamSesam was designed to be active for the whole day, for several days of the festival. That means that the busy and the non-eventful phases are all part of the sound sculpture’s musical output; it is a sound sculpture and thus performs for a longer duration than an individual would listen to. The extended duration of the musical component is one of the defining aspects of sound installations.³⁵ SesamSesam is a sound sculpture with installation qualities, namely the fact that it was installed for three days and nights, and was not accessible to the public afterwards. Additionally, it filled the whole entrance area including the staircase with music and defined a space in front (the sensor space) where the surrounding speakers gave the person in that area not only the best surround sound experience but also the possibility to interact with the sculpture’s music through bodily movement.

Potential and Limitations: Danced Sound Sculptures

To summarize, interactive sound sculptures do not equal musical instruments—that is what makes it possible for them to be danced. It is easier to achieve a musical result than with an instrument, but stylistically the musical possibilities are limited and accordingly predetermined.

The Terpsitone, the musical instrument that is played with full-body movement, cannot be danced with freely, since every dance gesture would mean losing control over the sound-producing instrumental gesture. The performer is limited to the ‘choreography’ of playing music.

Both Peter Vogel and Jutta Ravenna created their sculptures with audience interactivity as well as dancers in mind. Both worked with specific dancers for a dance performance: Vogel selected his musical material in accordance with each dancer and period; Ravenna adjusted reaction times and gesture-mapping according to the dancer’s suggestions. Both sound sculptures can be danced with. In these two cases, the dance style influences the selection of movement and sound structures in the development before the performance. The dancer can create a planned choreography.

The dancers also act as improvising musicians when dancing. In a way they are composing while they create their choreography. But the compositional aspect is limited to a choice from a pre-selection of material. The system-builder, in this case Vogel or Ravenna, composes the potential sounds—or even the potential music of the danced sound sculpture.