SoundWave Surfing

Gaming, Interactivity, and Contemporary Dance

John Toenjes

Aesthetic Bias about the Music and Dance Relationship

Since the early years of modern dance in America, composers and choreographers have been debating about the most preferred, most aesthetically valid relationship between music and dance. Some composers have said that the music should be closely tied together with the choreography. Others have claimed that the music should be a loose “frame” around the dance. Some have asserted that the music must be as interesting in itself as the choreography; some have insisted that music that could stand on its own aesthetically was not suitable for modern dance.¹ And later in the twentieth century, the radical idea arose that the music or a sound score could have no relationship with the dance at all, other than sharing the same time and space.

Many, if not most, contemporary modern dance artists subscribe to the idea that music should provide context for the choreography and perhaps some rhythmic impetus at times, but not illuminate the same information or illustrate the same picture that the dance does. In this relationship, each partner in the music-dance collaboration contributes something to which the other can respond. While this is certainly a valid point of view, it assumes that there is a difference between the music and the dance.

This isn’t necessarily the case in cultures that have no separate words for music and for dance.² But, in modern and contemporary western-European-based culture, the distinction is severe. What is the nature of the difference? Both music and dance are time-based arts, constructed along certain structural templates or concepts. Both communicate to the human spirit in non-verbal, non-literal ways. In both, bodily movement is the activator of the art. They both exist in space, as well. But here is where the fundamental difference lies: music’s space is the auditory, air space, and the dance’s space is the visual, earth space.

And what is the difference between musician and dancer? The choreographer José Limon delivered a convocation address at the Juilliard School, entitled “Dancers are Musicians are Dancers”³ that attempted to unite the two arts by illustrating how dance can be heard in the music of Bach, as dance ideas and forms were at the heart of his art. But on a more fundamental level, one can consider that a musician is creating music by blowing their breath through a tube or rubbing a bow back and forth on a string. Musicians are indeed moving while the music is being produced and heard. But because the musician’s movement is generally not as extravagant as the dancers, and usually is not thought of as having communicative value in and of itself, much of the separation happens physically, too, where the musicians are relegated to the “pit” or situated offstage.

However, this attitude is not grounded in evidence. According to researchers, musicians actually do participate in the expression of content through their movement. In a study published in the journal Psychology of Music, Mary Brougton and Catherine Stevens discovered that “expressive body movement plays an important role in the communication between […] performer and audience.”⁴ This supports the theory of “kinesthetic empathy” which has been extensively researched since it was introduced in the 1930’s by critic John Martin. Beyond kinesthetic empathy, Ryan Ingebritsen posits a concept of “auditory kinesthesia,” which is “a part of the body schema that involves a direct mapping between the auditory and sensorimotor system […] and can also contribute to kinesthetic empathy between audience and observer.”⁵

Orchestra conductors such as Nicolas McGegan⁶ move gracefully in a conducting “dance” in kinesthetic response to the music, which is mirrored back to the players in the orchestras he conducts and the audiences who watch and listen. This highly visible motor movement, both in response to and guiding the music making, moves towards the perception and conception of music and movement as one inseparable entity. And of course, in certain genres such as ballet folklorico, flamenco, and tap dance, the dancers make rhythmic sounds with their feet and/or hands, providing the music at the very same time as the movement. In these cases, there is no separation between dancer and musician.

And now, with computer motion capture and interactive media systems, the distinction between dancer and musician—even in forms that are not based on body rhythm—can be challenged and even erased. What if the music’s space is the aural/air space and the visual/ground space? Conversely, what if the dance’s space is the visual/ground space and the aural/air space, manipulatable through movement? In motion capture environments, the air space around a freely moving dancer is now the physical play space. The air space now feels tangible, physical. Then, similarly to the flamenco dancer, the computer-assisted dancer can truly be musician and dancer at the same time.

This is where my art resides in terms of the music/dance relationship: I generally want the dancer and the musician to be one and the same person, which I term a “movician.” With the tools available through computer technology, the new movician can have control over not only percussive rhythmic sounds, but also over traditional musical elements such as melody, timbre, texture, and musical structure, as well as more modern elements such as digital effects and audio space shaping, sampling, and the “mix.” My works, such as Leonardo’s Chimes, Songs of the Elastic Halloon, and SoundWave Surfing,⁷ are systems designed to create movicians through the construction of a musment—music + movement—environment. Leonardo’s Chimes is a movician game where the mover and audio controller are in a duet looping guessing game. Songs of the Elastic Halloon uses a musment improvisational environment to create a form of poetry that I call “recurring strophic,” a type of reiterative song-poetry that is constantly changing and reforming based upon the movement of a performer in the interactive environment.

I have heard complaints more than once from contemporary choreographers that they are tired of seeing someone in front of a camera “swooping” their arm, for example, and having the music “swoop,” too. This is an attitude born of the “framing” idea that doesn’t consider that contemporary artists at this moment are still trying to figure out just what the aesthetics of technology-enhanced movement art is. Given that they don’t rail against tap or flamenco dancers making sounds with their feet, I find complaints about a one-to-one mapping of sonic qualities to movement gesture puzzling. Much work is being done on gesture recognition to arrive at more sophisticated and multifaceted mappings of movement to music. As a matter of fact, gesture recognition machine learning algorithms are taking this field into the realm of sonification of the meaning behind gestural movement, rather than its measurable, physical patterning. This involves theories of the meaning of gesture, both within and across cultures, which is a deep and complicated realm of investigation ripe with possibility.

This attitude is lately being broken by other strong influences, particularly street/hip-hop culture. In this culture there is an interplay of rhythm and movement that is improvisatory and relates directly to the beat and to the rap recited along with the music. There is also a competitive aspect to it, and individuality and spontaneity are prized. Remix is central to this aesthetic as well. The current liberal emphasis on “decolonizing” art might just be breaking the prejudice against dance aligning closely with music’s rhythm and structure in the modernist aesthetic.

However, taking the criticism of those modernist choreographers seriously, what I do find aesthetically crucial is to become more sophisticated in the “mapping” of movement to sound. New possibilities in mapping allow for new conceptions in structure and effect and purpose in the art of musment.

Game Structure

Although the immersive tools used in arcade style and virtual reality video games have the potential to create movicians and musment, generally, commercial game designers are not taking advantage of this power to make a move toward creating movicians. Games such as “Dance Dance Revolution” and “Just Dance” use motion tracking to encourage imitative movement through keeping score of how well the player imitates the movement of an onscreen dancing avatar, or how closely the player can match the rhythm of a song with their movement. In these cases, motion tracking is a tool for teaching mimicry and rhythmic accuracy, but not for an iterative integration of creator and created. This is due in a large part to the need of a game to have a standard against which to keep score. Instead, the art dance/performance world is where we see the emphasis on integration developing, where score keeping is not the top priority.

In keeping with the theories of kinesthetic empathy and auditory kinesthesia mentioned above, one of my artistic missions is to engage the audience physically in various ways in movement-based interactive performance. Various works of mine have used different tactics in doing this. Critical Mass used our custom phone application LAIT⁸ to engage the audience in conversation and photo sharing around the performance. It allowed them to democratically change the course of the dance through voting and gave them license to come onstage to experience the performance in close proximity to the dancers, encouraging multiple perspectives on the stage set and its dynamic graphics.⁹ A later work, Alternate Reality, uses the mobile phone as a game controller and communication device to challenge the audience to work together to collectively solve riddles related to the concept of reality, and makes them travel to different spaces and even to learn a dance themselves to arrive at the complete answers.¹⁰

My latest interest in this regard is to create works that use game structures to engage a generation of people that grew up playing video games. Video games have become central to the experience of many people across the world. In Reality is Broken, a seminal book on the influence of video games in contemporary culture, author Jane McGonigal emphasizes that “97% of youth play computer and video games.”¹¹ The gaming lens can grab the attention of an audience in a more immersive way than traditional passive theater performances. I feel that the impact of video games on a changing society makes it crucial for artists to investigate their ability to drawing together the elements of “musment,” competition, social connections, immersive experience, and machine interactivity to make art in the twenty-first century.

According to McGonigal, essential to the definition of any game are the following elements: a goal, rules, a feedback system, and voluntary participation.¹² Competition and art have been strange bed fellows for centuries. Their goals are not neatly aligned. One values defeating an opponent or overcoming obstacles, the other prizes individual or collective expression of feelings or cultural values. Traditional performing arts have virtually none of the essential elements of a game. However, as digital media and computer games have begun shifting the idea of performance from a passive experience to an active model, where an audience can have agency over it, competition has found its way into the performing arts. This can be seen in a variety of arenas, such as in audience-approval shows like So You Think You Can Dance and in the power of choice that comes with the ability to quickly swipe through TikTok videos to create one’s own playlist of favorites.

In order create art that is also a game, the artist must ask themselves: what goal can I construct for my performers to reach? How do I construct rules that make the game compelling and give it structure? What is the feedback system or systems that the performers and/or audience will rely upon to feel like the art is progressing along a line of inquiry? Lastly, how do I get the audience to participate in the working out of this artwork?

SoundWave Surfing

SoundWave Surfing is an improvised “musidance” competition that requires the integration of “musment” and “movicians” within a game that is played by the performers and judged by the audience. This dance participates in and celebrates some aspects of hip-hop culture through its competitive nature, its emphasis on individuality and spontaneity, the freestyle movement and vocalizations, and the sampling and overdub/remix musical structure. This dance has gone through two iterations, one as a duet competition and one as a team competition. Both versions are similar in structure, but the team version involves the audience in deciding which team is the winner as judged by the enthusiasm of their response to the performances. In this writing, I will focus on the team performance.

The goal of the game is for the movicians to create an interesting mix of music and dance and to have the audience decide that one team or the other has achieved the better mix. As the emcee says to the audience in his opening script: “you’re not judging just how good the movement is, or how good the vocals are, but how well they come together in a new and innovative way.” The rules are that members of both teams, each comprised of three movicians, must both vocalize a completely spontaneous sonic challenge for their other team members to “surf,” and then to “surf” through their other team members’ vocalizations to contribute to the team remix. The feedback system is provided by a visualization of the sound wave with a shaded loop area that moves with them in real time, and of course, the audio mix in the speakers. Voluntary participation is a given, since both audience and performers have willingly come to the performance.

At the start of the performance, an emcee warms up the audience, instructing them to cheer for their preferred performers to determine the winners. This allows the computer programming to calibrate the microphone that is used to capture the audience response for each performance and cues the audience as to how to behave in the theater; cheering on your team is not usual behavior in an art dance context. After introducing the team members, the emcee demonstrates the workings of the technology to the audience. This is done because, as I wrote in my article “Composing for Interactive Dance,” “most artists agree that audiences tend to enjoy and understand interactive dance works better when they perceive the connection between the performers and the interactive system.”¹³ I have found through post-performance feedback sessions that audiences are still quite unfamiliar with such systems. Audiences are curious about how things work, and don’t fully give themselves over to a performance if the magic of how these computer-assisted improvisational environments work is too hard to comprehend. Explaining its functionality gives them insight into the production so they can better appreciate and enjoy it.

After this introduction, the competition gets underway. Each team takes a turn: one team member vocalizes a completely spontaneous and unrehearsed improvised phrase that is recorded by the computer and immediately begins playing back in a long loop. Another of the team members then enters onstage as the “surfer,” who dances through the sound wave and remixes a musical composition, their motion tracked by video cameras. Moving across from stage left to stage right and front to back, the surfer controls the tracking with the buttons on a Wii® controller they are carrying. The surfer’s position left to right determines the midpoint of the loop, and the perceived width of their body in the camera’s view determines the length of the loop. The volume of the track is controlled by the center position of the dancer upstage and downstage, tracked by an overhead camera.

Figure 1: A red team “movician” surfing the sound waves. © John Toenjes

As the movician surfs the sound wave, the sound wave and loop sections are projected in real time on the screen behind them for the audience to see. This involves the audience visually and aurally in what is going on in the looping functions. It’s a challenging task for a team member to come up with an engaging vocalization that adds to the audio that has already been created yet contributes something more that the surfer can work with. Once a suitable portion of the recorded challenge is isolated by movement, the dancer uses the Wii® controller to lock the sample into place. The sample then starts to loop, and the dancer can choose to dance with that sample, and then later again use the Wii® controller to unlock that loop and find another spot in the sound wave to loop. If there are already two or more recorded tracks looping, they can use the controller to scroll up and down through those tracks to unlock the loops that have already been established by other team members. A variety of musical textures can be created by unlocking and shifting loops among the various tracks, creating interesting mixes from the various vocalizations.

As the remixes progress, several other digital audio filters are available for the performers to use to manipulate sound quality. The filter parameters are controlled by the speed of the dancer’s movement and their location on the stage. This brings the list of the mappings of movement to music to:

• The loop start is mapped to the position of the dancer from left to right on the stage.

• The loop length of the sample being manipulated is mapped to the perceived width of the performer’s body.

• The volume of the sound sample is matched to the position of the dancer from upstage to downstage.

• Various parameters of sonic filters are mapped to positions from stage left to stage right and upstage and downstage, and to the perceived amount or speed of their movement.

The game consists of several rounds. In the first round, there are only two tracks for the team to remix. Instructions are given for them to vocalize in a somewhat melodic manner. There are certain musical parameters that seem to work better than others for this round. Melodies that quickly vary timbre and pitch give more opportunity for creating interesting loops than those that do so more slowly. In the second round, three tracks are vocalized for surfing, and the musical parameters are opened up to include more rhythmic utterances. The complexity of three tracks challenges the virtuosity of the movicians. In the third round, called “Anything Goes,” the movicians are instructed to vocalize anything they want, from melodic pitches to screams and guttural noises to nonsense poetry. This creates a wide and wild variety of tonal variations in the four-track mix.

The attentiveness to how one must move while paying instant close attention to both how the audio is sounding and to how it feels in their body demands a complete immersion into the experience by the movician. Much is lost internally if the movician drops out of that attentive state, and the audience can sense a lack of concentration. This is the primary virtuosity demanded of the SoundWave Surfing environment: nearly any movement or sound will work, but the integration of the music and movement is where the effort and attention lie.

When each team is finished with their mix, the audience is asked to cheer for their favorite, and the team scores are projected onto the screen. A final tally of all three rounds declares a winner at the end of each performance.

Audience Reactions

Young audiences seem to enjoy and appreciate this dance work. The dancers have fun, too, as it is exhilarating and challenging at the same time. There is also a sense of community among the entire population at the performance; movicians and audience participate in the experience together and the fourth wall is more than broken—it is shattered. A quote from a student audience member illustrates how it reached them:

[I] found everything new and fresh […] The first reason […] is that different from the formal type of dance performance, this dance is more like a competition, which is a form I have never seen before. The innovation of combining the concert performance with the gaming form is a part of the contemporary media culture, and this novelty makes me feel creative and unique about this dance. This form asks the dancers to interact with each other frequently and also interact with the audience. By doing this, audiences felt more engaged in this performance and had more fun at the same time. When I was watching this performance, I felt like I was also a member of this performance […] Also, in my perspective, this show inspired me to generate a new idea about the relationship between the music and dance.¹⁴

As a theatrical producer, I found myself walking a line between camp and art in this last production of SoundWave Surfing. I was attempting to create an engaging, entertaining “game show” ambiance to inspire enthusiasm and receptivity in the audience, all the while trying to keep the focus squarely on the artistic experience of the remix and the integration of the sound and movement, and the virtuosity required to achieve “movician” status. This is a balance that is important to weigh in this context.

More on the Feedback System

Ideally, the “movician” is physically and artistically reacting as quickly as possible—within milliseconds—to what they are hearing, integrating their movement with the sound in an immediate feedback loop, unifying sound and movement as closely as possible. How does the tracking mechanism affect the relationship between the mover and the sound they are hearing and creating? This interface is what manifests the relationship between mover and sound, so one must question how its design helps or interferes with the direct connection between sound perceived and movement executed and then sound perceived again. What role does the performer’s “intuition” play in this? Can the interface be designed in a way that increases the performer’s sensitivity to the immediate experience?

This design problem has been at the center of the various tracking and control mechanisms I have experimented with over the years in various interactive movement/music works. I have found that there needs to exist a balance between the complexity of the control system and the ability of the performer to utilize its functions while meeting the demands of performance. As accuracy and reliability of tracking systems have improved, I have found that a super-accurate tracking system encourages a level of trust that can dull the awareness of the moment. I have decided that I prefer a certain amount of variability and unpredictability—in other words, a certain amount of “slop”—in such systems. This forces the performers to be in constant dialog with the unexpected, listening at every moment for how their actions are affecting the sound.

In the team version of SoundWave Surfing, I pared down to a simple color-tracking system that draws a bounding box around the “surfer” which locates them in relation to the loop position and length, and I simplified the button choices on the Wii® controllers. To get a show onstage by opening night, I had to strip down the tracking to something the performers could handle. The demands of “intuitively” yet knowledgeably and judgmentally responding to what they were hearing and producing with their movement was great enough that if I added the extra burden of a complex tracking and control system, the “intuitive” connection was lost. It is simply quite demanding to attempt to completely integrate movement and music into one corporeal whole while improvising in real time and simultaneously controlling an interactive environment.

Ideally, however, movicians in this system would be able control more subtleties to create more sophisticated musical results. Musical parameters such as dynamic contours and tonal variations are not readily deduced from a simple bounding box. Thus, virtuosity in using a system with more control is needed to get more complex results. Certainly, there is an amount of practice involved in learning any system, and virtuosity can be achieved with an adequate amount of practice. But what sorts of multi-modal systems are best, and which are enough to not be so complex as to interfere with the spontaneity of a performer trying to integrate sound and movement into one being? Is there a limit to which the designer can expect any human to be able to respond in an immediate and intuitive way?

Another question arises: what is an accurate or useable mapping of movement to sound in terms of defining a standard of perceived movement and sound integration? Is the mapping where the modernist choreographer could be appeased or thrown off the track? And how does the mapping contribute to or harm the seemingly intuitive responses of the performer to the stimuli and media they control or find themselves immersed in?

Investigating these questions within the context of a performance piece is at the heart of using SoundWave Surfing as a tool for design study. A tracking system that forces the performer to constantly re-evaluate and recalibrate their body’s spatial relationship to sound is an effective tool for making a true movician. In this sense, SoundWave Surfing is a sensitivity practice tool, a musical instrument, a game, and a dance environment. To SoundWave Surf, means to integrate all these aspects into an engaging and aesthetically valid staged game/art performance.

---

1. John Toenjes, “Musicians of the Movement: The ‘Composer-Accompanist’ in the Formative Years of Modern Dance,” in International Guild of Musicians in Dance Journal, vol. 3 (1994): 24-39.↩︎

2. Jerome Lewis, “A Cross-Cultural Perspective on the Significance of Music and Dance on Culture and Society, with Insight from BaYaka Pygmies,” in Language, Music and the Brain: A Mysterious Relationship, ed. Michael Arbib, Strüngmann Forum Reports, vol. 10 (Massachusetts: MIT Press, 2013).↩︎

3. José Limon, Convocation address, Juilliard School, October 5, 1966.↩︎

4. Mary Broughton and Catherine Stevens, “Music, Movement and Marimba: an Investigation of the Role of Movement and Gesture in Communicating Musical Expression to an Audience,” in Psychology of Music, vol. 37 (April 2009): 137-53.↩︎

5. Ryan Ingebritsen, Auditory Kinesthesia: A Framework to Facilitate the Design and Performance of Interactive Music Systems (PhD dissertation, University of Illinois, 2022, unpublished), 32.↩︎

6. Conductor of the Philharmonia Baroque Orchestra among other orchestras around the world.↩︎

7. These can be seen at https://sites.google.com/jtoenjes.com/cv-works/selected-work-samples/interactive-dance (accessed 16 July 2022).↩︎

8. “LAIT—The Laboratory for Audience Interactive Technologies: Don’t ‘Turn it Off’—‘Turn it On!’ Proceedings of the 21st International Symposium on Electronic Art,” accessed 16 July 2022, http://www.isea-archives.org/docs/2015/proceedings/ISEA2015_proceedings.pdf, 334-7.↩︎

9. https://sites.google.com/jtoenjes.com/cv-works/selected-work-samples/interactive-dance (accessed 16 July 2022).↩︎

10. https://sites.google.com/jtoenjes.com/cv-works/selected-work-samples/interactive-dance (accessed 16 July 2022).↩︎

11. Susan McGonigal, Reality Is Broken: Why Games Make Us Better and How They Can Change the World (New York: Penguin Press, 2011), 22.↩︎

12. Ibid., 30.↩︎

13. John Toenjes, “Composing for Interactive Dance,” in Perspectives of New Music, vol. 45, no. 2 (Summer 2007): 28-50, 37.↩︎

14. Excerpt of a school paper, University of Illinois, Department of Dance, 2022.↩︎