InfoMus Lab
Mapping of expressive gestures
A main research issue in our investigation is the study of the
mapping strategies of recognized expressive content onto multimodal outputs,
i.e. the exploration about how the expressiveness conveyed by expressive
gesture can influence the generation of multimodal (sound, music, lights,
movements, visual media) outputs.
Experiments start from immediate cause-effect relations to move toward
long-term complex mapping including dynamic molding of the environment and
expressive multimodal gesture of character and avatars.
The following figures show some examples of mapping of expressive cues onto visual output. The figures refers to real performances organized in the framework of the EU-IST Project MEGA. The EyesWeb open platform was employed for extraction of expressive cues, mapping, and visualization.
 |
| Deformation of an image depending on expressive audio and motion cues. |
The figure above shows deformations of an image depending on expressive audio and motion cues. A lens metaphor is used. Different kinds of deforming lenses are available in EyesWeb. The amount of deformation is related to values of the extracted cues (e.g., Quantity of Motion, Contraction Index.) The dancer's body, the background or both of them can be deformed. For example, in the left image the background is deformed. The dancer's body is instead deformed in the image on the right. The background bitmap can be selected by the artist and can be dynamically changed during the performance.
In the EyesWeb patch displayed in the figure here below the word "Gesture" is processed in real-time and moved on the screen depending on the motion cues extracted during a dance performance (the processed silhouette of the dancer is also shown in the output image). Several motion cues are used: for example, overall motion direction is used to control the motion of the word on the screen, Quantity of Motion is mapped on the color of the word, movement is segmented in pause and motion phases and the actual phase is used to switch among two different words: thus, the word "Expressive" is displayed during pauses, while the word "Gesture" is displayed when the dancer is moving. The amount of shadow and its color, both of the word and of the dancer's silhouette can also change in real-time depending on motion and audio cues. Motion cues are extracted with the EyesWeb Gesture Processing Library.
 |
| The word "Gesture" is processed and moved in real-time depending on values of motion cues. |
A mapping among extracted motion cues and the movement of a robot on-stage was developed for a robot-dancer dialog in the framework of the performance "L'Ala dei Sensi" (Ferrara, Italy, November 1999).
Main references
A. Camurri, B. Mazzarino, G. Volpe
Expressive interfaces, Cognition, Technology & Work, Springer-Verlag,
Published on line, December 2003.
A. Camurri, B. Mazzarino, G. Volpe, P Morasso, F.
Priano, C. Re
Application of multimedia techniques in the physical
rehabilitation of Parkinson's patients, Journal of Visualization and
Computer Animation, 14(5), pp. 269-278, December 2003.
A. Camurri, P. Coletta, M. Ricchetti, G. Volpe
Expressiveness and Physicality in Interaction, Journal of New Music
Research, Vol. 29, No. 3, pp. 187-198, Swets & Zeitlinger, September 2000.
A.Camurri, P.Coletta, M.Ricchetti, G.Volpe
Synthesis of Expressive Movement, in Proc. Intl. Conf. ICMC-2000,
Berlin, Germany, August 2000.