InfoMus Lab |
Multimodal Expressive Environments
A main objective of our research activity is to give a scientific and technological contribution to the development of novel forms of artistic performances, where the performing action takes place in a number of interactive physical as well as virtual connected spaces. In these spaces spectators become participants, since they are enabled to directly generate and modify content through interaction. To this aim we developed models of multimodal expressive environments to be employed as a reference framework in the design of performance spaces aiming at enhancing the experience and the sense of presence of participants. Such models include Multimodal Environments (MEs) and the more recent Extended Multimodal Environements (EMEs) and Multilayered Integrated Expressive Environments (MIEEs). Multimodal Environments (MEs) are systems capable of establishing creative, multimodal user interaction, by exhibiting real-time adaptive behavior. In a typical scenario, one or more users are immersed in an environment allowing them to communicate by means of body movement, including dance and gesture, and possibly by singing, playing. Users get feedback from the environment in real time in terms of sound, music, visual media, and changes in the world by means of actuators in general, including on-stage navigation of semi-autonomous mobile systems. MEs can be considered as agents capable to observe the users, extract information about their expressive gestures, and change their reactions, their "social interaction" and rules over time: a gesture of a user can mean different things in different situations. To this aim, MEs should embed multi-level representations of different media and modalities, as well as representations of communication metaphors, of emoting potentials. With Extended Multimodal Environements (EMEs), the concept of ME is specified in more details from the one hand, and further extended on the other hand (i) by explicitly including humans (usually, performers and spectators/participants) in the model and (ii) by explicitly envisaging contexts in which the performance is spread over a number of distributed physical and virtual spaces together constituting a shared performing environment. Extended Multimodal Environments (EMEs), conceived as Mixed Reality spaces containing real and virtual objects and real and virtual subjects, represent the basic bricks of a Multilayered Integrated Expressive Environment (MIEE). MIEEs are multilayered since they represent a performance with respect to narrative structures situated at several layers of abstraction. They are integrated since a number of particular aspects of the interactive performance, such as analysis of spectators/participants' behavior, real-time generation of multimedia output, individuation and application of suitable mappings between analyzed behavior and generated output, management of the whole performance at multiple layers are all grouped and considered under the same conceptual framework. They are expressive since most of the interaction and communication processes taking place inside them are aimed at conveying expressive, emotional, affective content. Multimodal expressive environments open new niches of applications, many still to be discovered, including music, dance, theater, museal exhibitions, interactive arts, entertainment, edutainment, information atelier, and other industrial applications. For example such environments can be employed in a discotheque, a theatre, a museum, a school, a rehabilitation center for patients with a variety of sensory/motor and cognitive impairments. In fact, multimodal expressive environments generalize the bio-feedback methods which already have found widespread applications. An exampleLet us consider a simple concrete example of a ME as a "living agent", capable to extract from a dancer some gesture and motion aspects, thereby controlling the generation of sound and music. At the beginning, the ME is a "tabula rasa", nothing is evoked by movement: the system is observing the user. We can imagine that the ME is trying to identify features of the movement style of the dancer: if he/she/it starts moving with nervous and rhythmic gestures in roughly fixed positions in the space, therefore evoking the gestures of a percussionist, the ME, after a few seconds, initiates a continuous transformation toward a set of virtual percussions, located in points of the space where the dancer insists with her movement. "Continuous" means for example that neutral sounds begin to emerge and transform progressively into percussions, e.g., gradually reducing the attack duration and moving from a default to a specific timbre. The number, the sound characteristics, and the spatial position where the percussions are located is decided by the movement of the dancer. At this point, the dancer is allowed to play the instrument she has built. Instruments not played for a certain period of time may begin to fade away. As soon as the dancer changes her "style" of movement, e.g., by reducing the force impact, her energy, toward a "smooth", wider gesture, the MEs will follow such change and adapt to another context (again, continuously and in a time interval proportional to the amount of change of the dancer style of movement). This transformation means a continuous change both in the sensitivity and focus of attention to movement and gesture, and in the associations and rules regarding the sound output and music database (of fragments, rules, etc.). Music fragments can be either pre-existent or generated ad hoc in real time from the system, coherently with the current new gesture family and movements and the music composition goals. In our example, the transformation might change the music output from the set of user-defined virtual percussions into a (virtual) string quartet where the movement controls the interpretation and the melodic contour of the counterpoint. The system generates the counterpoint and is subject to compositional goals (thus incorporating a deliberative behavior). The designer of the performance (the director/composer/choreographer) has to put into the system the knowledge, the rules of change between contexts, the goals, etc., and may leave to the system degrees of freedom on generative and compositional choices. Real applications based on these ideas have been developed and are currently experimented with our systems by various artists.
Main references A. Camurri, P. Ferentino
(1999) A. Camurri (1997)
|