IMEJ main Wake Forest University Homepage Search articles Archived volumes Table of Content of this issue

1. Introduction
2. Performance Design
2.1 Space & Sound Design
2.2 Interpretative Possibilities
2.3 Poe's Poetics
3. Implementation & Evaluation
3.1 Staging
3.2 Testing
4. Conclusions & Future Work
5. References


Printer friendly version

Terra Incognita: A Multimedia and Robotics Interactive Performance Centered around Edgar Allan Poe's "Dream-Land"
Nikitas M. Sgouros & Sophia Kousidou, University of Piraeus
Yiannis Melanitis, Performance Artist

Rapid advances in multimedia and robotic technologies have created a rich set of opportunities for incorporating new means of expression in interactive performances of literary works. This paper describes Terra Incognita, a Java-based project that explores the interaction between various multimedia objects, robotic actors and the audience in an interactive, poetry performance of Edgar Allan Poe's "Dream-land." Based on the poem's text, the project creates an "unidentified" space and invites the audience to explore it. Terra Incognita seeks to transform the reading of the poem from an internal, individualized process to a situated, social activity that is shared and shaped by the performance audience. The paper explains the design of the performance, describes its technical implementation, and gives information on its public staging.

1. Introduction
Rapid advances in multimedia systems and robotics greatly enhance the means of expression that are available in interactive performances of literary works. In particular, the use of robots in these events enables artists or educators to exploit the expressive potential of physical space and to provide an audience with novel ways of interacting with multimedia content.

While multimedia renditions of literary works have achieved significant popularity mainly through the creation of CD-ROM versions of various books, there has been little interest in integrating non-traditional media types such as robots or mixed reality environments in these projects. A notable exception concerns the development of advanced narrative environments such as robotic storytellers [Druin 1999] or interactive playspaces [Bobick 2000].

This paper attempts to fill this gap by describing Terra Incognita, a Java-based project that explores the interaction between various multimedia objects, robotic agents, and the audience in an interactive poetry performance of Edgar Allan Poe's "Dream-Land." In Section 2, we explain the performance design of this event. Section 3 presents the technical realization of Terra Incognita and gives information on its public staging and testing. Finally, section 4 is a "conclusions and future work" section. An appendix contains the text of Edgar Allan Poe's "Dream-Land."

About the authors...

2. Performance Design
Terra Incognita consists of a mobile robot hosting a sculptural form of a six-week human embryo that moves inside an "unknown" environment. This environment also hosts a second robot equipped with a video camera that can be tele-operated by the audience. The signal from the camera is broadcasted to both the robot operator, who may be any audience member, and to a video projector that shows a large-scale version of the camera input and the user interface. While controlling the robot, the audience hears a robotic voice generated from a speech synthesizer that recites "Dream-Land," one line at a time. The system displays the text associated with the recitation of each line in a small area on the screen. The system uses two voices for recitation, a male and a female one. The order in which these voices are heard is random. Furthermore, the time between successive recitations is random.

Apart from the recitation, the audience is exposed to real-time audio effects that are generated constantly. These effects mimic the static noise of a short-wave radio transmission. In addition, the audience hears MIDI sounds that are generated by a composing algorithm linked to the movement detected by the video camera onboard the tele-operated robot.

The user controls the camera-bearing robot using a set of four arrow buttons on the user interface window. Using these buttons an audience member can instruct the robot to move forwards, backwards, turn left or right. The movement of the embryo-carrying robot is autonomous and random in the four directions described above.

2.1 Space and Sound Design
At a conceptual level, Terra Incognita seeks to create an 'unidentified' form of physical space representing a 'Terra Incognita' of art and invites the audience to explore it. The performance seeks to create a physical space that possesses the basic qualities of the 'land' described in the poem: dark, alien, harsh, dream-like and unattainable.

To this end, both robots move in a flat, empty area, significantly larger than their size. This area is lit using only two spotlights located at ground level in a direction parallel to the ground. The lighting design seeks to create an unnatural form of ambient illumination. (In real life light comes from above and not from the side as in this work.) The size of the space, its darkness, and the unnatural lighting conditions give the impression of an alien space when viewed from the robot camera. This impression is reinforced by the artificiality and harshness of the voices used in the recitation of the poem.

By the use of MIDI sounds, we seek to impose the dream-like qualities of the space described by Poe. The instruments that correspond to these sounds are selected from a statistical process that favours piano or percussion sounds over brass or string instruments and low notes over high ones. Consequently, this process creates dynamically a soundtrack inspired by some of John Cage's piano works [Cage]. The frequency with which these sounds are generated is proportional to the amount of movement detected by the robotic camera. Movement is detected whenever the embryo-carrying robot moves in front of the camera or when the camera-bearing robot executes instructions from the user. As a result, the density with which MIDI sounds are produced is proportional to the level of activity in the event and this reinforces the sense of control that the audience has over the content of the performance.

Finally, the continuous generation of noise-like audio effects similar to the noise heard in short-wave radio transmissions serves to impose the sense of unattainability of the 'land' described by Poe. An unattainable space cannot be analyzed or broken down into constituent parts; thus, all possibilities are left open during its exploration. In a similar vein, noise is a 'gestalt' experience containing all frequencies and, thus, all possibilities for expression. Furthermore, the dark and passive embryo-form that is carried around in space by a machine provides a concrete metaphor for the human inability to experience and actively explore the 'land' in the poem.

In order to capture the sense of awe and impotence that Poe feels towards the space he describes in "Dream-land," the performance is structured in a contrapuntal manner. The existence of parallel, contrasting elements at the interaction level (an active, user-controlled robot versus a passive sculptural form that is being moved around in a random fashion and remains indifferent to the user's actions) achieves a structural tension that emphasizes the futility of the audience's efforts to understand the mysteries of this land. This tension is further emphasized by the juxtaposition of structured MIDI sounds and noise during the performance, along with the use of a male and a female voice in the poem's recitation.

2.2 Interpretative Possibilities
Terra Incognita attempts to establish possibilities of meaning in a variety of levels during the event, thus emphasizing the inherent polysemy of Poe's poetic language. The performance becomes dynamic and malleable, which allows the user to explore, at her/his own pace, different interpretative possibilities of the literary work and react to what s/he perceives. Therefore, Terra Incognita attempts to enrich our interpretation experience of the poem with a spatial awareness that transcends the constraints of printed or, more generally, audiovisual media. The interpretation of the poem during the performance becomes an active, observable process in physical space that can be shared by the audience and can instigate more intense exploration of the literary work.

The use of random MIDI sounds and noise during the performance along with the random movements of the robot carrying the sculptural form is influenced by the performance aesthetics proposed in the works of John Cage and Merce Cunningham. Static radio noise and the use of chance procedures in generating musical phrases that are linked to audience activity are inspired by Cage's use of everyday sounds as musical instruments, his 'non-intentional music' forms and his desire to create aural landscapes that make it clear to the listener that the hearing of music is a product of his/her own actions, that the music is his/hers rather than the composer's. Furthermore, the use of random movements in the behaviour of the sculpture-carrying robot that are disassociated from both the soundtrack and the narrative process are inspired by Cunningham's choreographies and seek to establish pure movement as one of the primary forces in generating tension and drama during the performance.

2.3 Poe's Poetics
Poe's poetic work was specifically chosen as the main subject of this performance because of his belief that poetry should be used as a means for the 'contemplation of the beautiful' (Poe, p. 1376). According to Poe beauty manifests itself in literature through the generation of 'impressions' or 'effects' that induce strong affective reactions in the reader, with 'melancholy [being] the most legitimate of all the poetical tones' (p. 1377). Furthermore, Poe believed in the immediacy of the reading experience and was sceptical towards the creation of literary works that could not be read at one sitting since the vastly important effect of the totality or unity of effect would be destroyed in these works (p. 1375). Terra Incognita seeks to integrate the presentation of the poem in a perceptual environment that enhances the affective qualities of "Dream-Land" and leads the user to a state of excitement or elevation in line with the poet's goals.


A QuickTime movie (~1 MB): A second robot is equipped with a video camera that can be tele-operated by the audience.

A QuickTime movie (~1 MB): The signal from the camera is broadcasted to both the robot operator, who may be any audience member, and to a video projector that shows a large-scale version of the camera input and the user interface.


3. Implementation and Evaluation
The technical implementation of this work reflects our belief that robotic and multimedia performances of literary works can find wide application only if they are realized in cost-effective ways. To this end, the robotic hardware used in the performance consists of two easily assembled robotic kits from Lynxmotion (each one costing around $250). The rest of the equipment consists of a general purpose PC and a cheap Web camera. All the software that was developed for the performance was based on the Java platform, which is freely available on the Web.



Lynxmotion web site.

Java web site.

3.1 Staging
The first staging of this work took place at the Decima Muestra Internacional de Performance: La Vida en Otro Planeta, Ex Teresa Arte Actual, Mexico City, October 11-21, 2001. Terra Incognita received a warm reception from the audience. More specifically, the audience was integrated successfully in the development of the event and found its interaction with the system meaningful and enjoyable. Although the original design of the performance specified that the environment hosting the two robots would not be visible to the audience except through the signal sent from the video camera, this was not possible due to space restrictions in the exhibition space. However, this did not limit the effectiveness of the performance. In fact, providing access to the robot space increased the interest of the audience in participating in the event.

3.2 Testing
In order to assess more clearly the effect that the performance had on the audience's experience of the poem, a set of informal trials involving six users took place at the University of Piraeus. The users were asked to first read 'Dream-Land" from a book and then participate in Terra Incognita. Finally, they were asked to compare their experiences in terms of the level of arousal, playfulness and engagement that characterized each activity. In terms of arousal all of the users agreed that they felt more excited during Terra Incognita. Their prevailing mood was that of alienation and awe. These reactions were consistent with the mood of the poem and the goals of the performance. In terms of playfulness and engagement, the users thought that Terra Incognita provided a more open-ended, challenging, and engaging experience than did a traditional reading of the poem.

Furthermore, the users were asked to describe the centrality of the poem in the Terra Incognita experience. Four of the users thought that the poem was really central to the event. They paid attention to the content of the poem in order to find clues that could aid them in structuring and interpreting the event. The remaining two users focused more on the exploration of the performance space and less on reading or listening to the text of the poem. These users felt that they would have paid more attention to the "Dream-Land" content if the text had been presented as part of the video stream and not at a separate area at the bottom of the video window. All of the users, none of whom were native English speakers, complained that the articulation of the synthetic voices was sometimes unclear. In these cases they had to resort to the text window to fully understand the verse.

Two video footage from the event:
QuickTime movie (~900 KB)
QuickTime (~670 KB)

4. Conclusions and Future Work
This paper describes Terra Incognita, a multimedia and robotics interactive performance of Edgar Allan Poe's "Dream-Land." Terra Incognita seeks to transform the reading of the poem from an internal, individualized process to a situated, social activity that is shared and shaped by the performance audience.

Future work in this area will focus on the creation of more complex interactive performances of literary works. These works will feature robotic actors with highly sophisticated behavioural repertoires that can adapt, in real-time, their individual or collective behaviour to achieve synchronization with various multimedia objects and coordinate their interaction with the audience based on higher level plot structures [Sgouros 2000, 1999]. To this end, we are currently developing a new generation of authoring and execution environments for the scripting, direction and control of robotics and multimedia performances [Sgouros 2001].

5. References
Cage, John. Complete Piano Music vol. 3 "Music of Change.s" Audio CD, M•DG Series (S. Schleiermacher - piano).

Sgouros, N. M., and Kousidou, S. (2001). Generation and Implementation of Mixed-Reality Narrative Performances Involving Robotic Actors. In Virtual Storytelling. Using Virtual Reality Technologies for Storytelling. O. Balet, G. Subsol, P. Torguet (Eds.). Lecture Notes in Computer Science, vol. 2197, Springer.

Druin, A. et al. (1999). Designing PETS: A Personal Electronic Teller of Stories. Proceedings of CHI'99. ACM Press.

Bobick, A. F., et al. (March 2000). The KidsRoom. Communications of the ACM. 43 (3).

Sgouros, N. M. (November 2000). "Detection, Analysis and Rendering of Audience Reactions in Distributed Multimedia Performances." ACM Multimedia 2000, Los Angeles, CA, 195-200.

Sgouros, N. M. (1999). "Dynamic Generation, Management and Resolution of Interactive Plots." Artificial Intelligence. 107 (1), 29-62.

Poe, E. A. (1996). "Dream-Land" and "The Philosophy of Composition." Poe: Poetry, Tales, and Selected Essays. New York: Library of American Colleges Editions.

********** End of Document **********

IMEJ multimedia team member assigned to this paper Yue-Ling Wong