InfoMus Lab
Laboratorio di Informatica Musicale
Expressive gesture as conveyor of emotion in an audience
An important topic for our research is the investigation of expressive gesture as a way to induce intense emotional experience in the audience.
A preliminary study has been carried out on this issue. It consisted of an experiment investigating the mechanisms responsible for the audience's engagement in a musical performance. The aim of this experiment was twofold:
- Individuating which auditory and visual cues are mostly involved in conveying the performer's expressive intentions
- Testing the developed model by comparing its performance with spectators' ratings of the same musical performances.
A professional concert pianist (Massimiliano Damerini) performed Etude Op.8
No.11 by Alexandr Skriabin at a concert that was organized for the
experiment's purpose at
the site of the Lab at Opera House Carlo Felice. He performed the
piece first without public in a normal manner and an exaggerated manner, and
then with public in a normal, concert manner. Exaggerated
means in this case with an increased emphasis in expressiveness consistent with
the style of performance of early 20th Century pianist style.
The
pianist performed on a grand coda piano (Yamaha Disklavier), which made it
possible to register MIDI information of the performance. In addition, we did
audio and video recordings from four sides. The video recordings from the left
were presented to the participants of the experiment.
 |
 |
 |
 |
| Video recordings of the piano performances (left, top, right, and front views) | |
Twelve students participated in the experiment; among them were four musicians. The participants saw the performances on a computer screen and heard them over high-quality (Genelec) loudspeakers twice. At the first hearing, they indicated the phrase boundaries in the music by pressing the button of a joystick. At the second hearing, they indicated to what extent they were emotionally engaged with the music by moving a MIDI-slider up and down.
Expressive cues were extracted from audio, video, and MIDI recordings. They included cues related to the movement of the head of the pianist (e.g., kinematical cues) from video, key-velocity and onset times from MIDI, cues related to loudness, roughness, rhythmic aspects from audio.
The
analyses of performance data suggested an opposite relation between emotional
intensity and the performer's posture. The pianist leaned forward for softer
passages and backward for intensive passages. In addition it suggested a
differentiation in expressive means with tempo on one side and key-velocity and
movement velocity on the other side.
InfoMus staff involved: A. Camurri, R. Dillon, M. Ricchetti, G. Volpe
Visiting researchers: R. Timmers, M. Marolt
Master Thesis students: R. Casazza, A. Pertino
Thanks to Pianoforti Storti for availability of Grand Coda Yamaha Disklavier.
Main references
A.
Camurri, B. Mazzarino, M. Ricchetti, R. Timmers, G. Volpe
Multimodal analysis of expressive gesture in music and dance performances,
in A. Camurri, G. Volpe (Eds.), "Gesture-based Communication in
Human-Computer Interaction", LNAI 2915, Springer Verlag, 2004.
R.
Timmers, A. Camurri, G. Volpe
Expressive gestures and their relation to emotional engagement in three
performances of a Skriabin Etude, in Proc. International Conference Music
and Gesture, Norwich, UK, August 2003
R.
Timmers, A. Camurri, G. Volpe
Performance cues for listeners' emotional engagement, in Proc.
nternational Stockholm Acoustic Conference 2003 (SMAC03), Stockholm, Sweden,
August 2003.
R.
Timmers, A. Camurri, G. Volpe
The expressive functioning of two acoustic cues in three performances of a
Skriabin Etude, in Proc. XIV Colloquium on Musical Informatics, Firenze,
Italy, May 2003.