Emotion-Oriented Systems: The Humaine Handbook (2011) .. edited by Paolo Petta, Catherine Pelachaud, Roddy Cowie
Table of Contents
Preface … xiii
PART 1: FOUNDATIONS … 1
Chapter 1. Contemporary Theories and Concepts in the Psychology of Emotions … 3
Géraldine COPPIN and David SANDER
1.1. Introduction … 3
1.2. Emergence of a scientific approach to emotions … 4
1.2.1. The emotional sequence: James-Lange versus Cannon-Bard … 4
1.2.2. Schachter’s two-factor theory … 6
1.3. Basic emotions theories … 7
1.3.1. Premises of basic emotions theories … 7
1.3.2. Characteristics of basic emotions … 8
1.3.3. Criticisms of basic emotions theories … 10
1.4. Bi-dimensional theories of emotion … 11
1.4.1. Premises of bi-dimensional theories of emotion … 11
1.4.2. Criticisms of bi-dimensional theories of emotion … 12
1.5. Appraisal theories of emotions … 14
1.5.1. Premises of appraisal theories of emotion … 14
1.5.2. Specific models of this theoretical trend … 15
1.5.3. Criticisms of appraisal theories of emotion … 18
1.6. Conclusion … 19
1.7. Glossary … 20
1.8. Bibliography … 21
Chapter 2. Emotion and the Brain … 33
Andy CHRISTEN and Didier GRANDJEAN
2.1. Introduction … 33
2.1.1. Emotions and the brain: the emergence of affective neuroscience as an independent discipline … 34
2.2. The major role of affective neuroscience in understanding emotions … 35
2.2.1. Emotion and the brain: from a unitary entity to processing, from structure to neural networks … 36
2.2.2. Levels of processing in emotional processes … 38
2.2.3. Emotion and cognition … 39
2.3. The historical and conceptual legacy of early conceptions of emotions and the brain … 40
2.3.1. Forerunners of affective neuroscience … 40
2.3.1.1. Charles Darwin … 40
2.3.1.2. The James-Lange peripheralist theory … 41
2.4. Initial neuro-anatomical emotion theories … 41
2.4.1. Canon-Bard’s centralist theory … 41
2.4.2. Papez’s circuit … 42
2.4.3. MacLean’s limbic theory … 43
2.5. Structures in the brain and their functions in emotional processes … 44
2.5.1. Amygdala … 44
2.5.2. Amygdala and emotional learning processes … 45
2.5.2.1. Amygdala and classical conditioning … 45
2.5.2.2. The amygdala: a structure with two processing streams … 46
2.5.3. The amygdala and emotional perception: hypotheses around the specificity of processing within the amygdala … 48
2.5.3.1. The amygdala as a fear module … 48
2.5.3.2. The amygdala and arousal … 50
2.5.3.3. The amygdala as a relevance detector … 51
2.5.4. The amygdala and memory processing … 52
2.6. The prefrontal cortex … 53
2.6.1. The prefrontal cortex and bodily signals … 55
2.6.2. The prefrontal cortex and the top-down regulation of behavior … 56
2.6.3. The prefrontal cortex and the motivational component of emotion … 57
2.7. The anterior cingulate cortex … 58
2.8. The role of the insula in disgust … 58
2.9. Temporal dynamic of brain processes in emotional genesis … 59
2.10. Functional connectivity … 60
2.10.1. Investigations of the connectivity using brain imaging techniques (MRI) … 60
2.10.2. Investigations into connectivity using electroencephalographic (EEG) techniques … 61
2.10.3. Benefits of brain connectivity studies … 62
2.11. Conclusion … 63
2.12. Bibliography … 64
PART 2: NON-VERBAL BEHAVIOR … 77
Chapter 3. Emotional Corpora: from Acquisition to Modeling … 79
Laurence DEVILLERS and Jean-Claude MARTIN
3.1. Introduction … 79
3.2. Building corpora: “acted”, “induced” and real-life emotions … 81
3.2.1. Acted data … 82
3.2.2. Induced data … 83
3.2.3. Real-life data … 84
3.2.4. Comparison of different types of data … 85
3.3. Current emotional corpora … 86
3.4. Coding schemes … 86
3.4.1. Emotional annotation protocols … 89
3.4.1.1. Definitions of annotated features … 89
3.4.1.2. Annotation … 90
3.4.2. Annotating context … 91
3.4.2.1. Validation … 92
3.5. Complex emotions in spontaneous data … 93
3.6. Applications for corpora … 97
3.6.1. Detecting and deciphering emotions in speech … 97
3.6.2. Designing an expressive agent from corpora … 98
3.7. Conclusion … 100
3.8. Bibliography … 101
Chapter 4. Visual Emotion Recognition: Status and Key Issues … 107
Alice CAPLIER
4.1. Introduction … 107
4.2. What is a facial expression? … 109
4.2.1. Definition … 109
4.2.2. Description … 109
4.2.3. Ekman’s universal expressions … 111
4.2.4. An ideal system … 112
4.3. Overview of facial expression recognition methods … 112
4.3.1. Databases … 112
4.3.2. Preprocessing: extracting the face … 114
4.3.3. Extracting facial characteristics … 115
4.3.4. Classification … 117
4.3.5. Performance … 117
4.4. Spontaneous facial expressions … 118
4.4.1. Position of the problem … 118
4.4.2. Databases … 119
4.4.3. Recognizing spontaneous expressions … 122
4.5. Expression intensity … 124
4.6. Dynamic analysis … 126
4.7. Multimodality … 128
4.8. Conclusion … 131
4.9. Bibliography … 132
Chapter 5. Recognition of Acoustic Emotion … 139
Chloé CLAVEL and Gaël RICHARD
5.1. Introduction … 139
5.2. Principles of automatic emotion-recognition systems … 140
5.3. Acoustic descriptors … 141
5.3.1. Voiced versus unvoiced content … 142
5.3.2. A temporal unit for emotional analysis … 143
5.3.3. Prosodic descriptors … 143
5.3.3.1. Fundamental frequency (pitch) … 144
5.3.3.2. Intensity … 145
5.3.3.3. Rhythm descriptors … 146
5.3.4. Voice quality descriptors … 146
5.3.4.1. Normalized amplitude quotient … 147
5.3.4.2. Frequency modulation (jitter) … 147
5.3.4.3. Amplitude modulation (shimmer) … 148
5.3.4.4. Rate of unvoiced windows … 148
5.3.4.5. Harmonic to noise ration … 148
5.3.5. Cepstral and spectral descriptors … 149
5.3.5.1. Formant parameters … 149
5.3.5.2. Mel-frequency cepstral coefficients … 149
5.3.5.3. Bark band energy … 150
5.3.5.4. Spectral centroide … 151
5.4. Automatic emotion classification … 151
5.4.1. Choosing descriptors … 152
5.4.1.1. Normalizing descriptors … 152
5.4.1.2. Reduction in data representation space … 152
5.4.2. Learning algorithms … 154
5.4.2.1. Separators with vast margin … 154
5.4.2.2. Gaussian mixture models … 155
5.5. Performance and assessment … 157
5.5.1. First factor: data and classes of emotions … 157
5.5.2. Second factor: the problem of “ground truths” … 159
5.5.3. Third factor: manual preprocessing … 160
5.5.4. Fourth factor: learning algorithms … 160
5.5.5. Fifth factor: learning conditions … 161
5.6. Conclusion … 161
5.7. Bibliography … 163
Chapter 6. Modeling Facial Expressions of Emotions … 169
Sylwia Julia HYNIEWSKA, Radoslaw NIEWIADOMSKI and Catherine PELACHAUD
6.1. Expressive conversational agents … 169
6.2. Expressions and their emotional states … 170
6.2.1. Expressing discrete emotions … 171
6.2.2. Dimensional approaches to emotional expression … 171
6.2.3. Componential expression of emotions … 173
6.3. Computational models for facial expressions of emotions … 174
6.3.1. A discrete representation of facial expressions … 174
6.3.2. Dimensional representation of facial expressions … 174
6.3.3. Componential approaches to facial expressions … 176
6.3.4. Mixtures of emotions and social constraints … 178
6.3.5. Sequences of emotional expressions … 181
6.4. Conclusion … 183
6.5. Acknowledgements … 184
6.6. Bibliography … 184
Chapter 7. Emotion Perception and Recognition … 191
Ioana VASILESCU
7.1. Introduction … 191
7.2. Perception in vocal communication of emotion … 193
7.3. Experimental paradigms and emotion-oriented automatic systems … 194
7.3.1. Experiments validating emotional content and/or annotation strategies … 195
7.3.1.1. Experimental design and stimuli … 195
7.3.1.2. Test populations … 196
7.3.1.3. Perceptual categorization into emotional classes … 196
7.3.1.4. Choosing emotional labels … 197
7.3.1.5. Discussion … 199
7.3.2. Tests for validating measurable parameters of emotional information … 202
7.3.2.1. Discussion … 205
7.3.3. Tests comparing human and automated emotion recognition … 205
7.3.3.1. Discussion … 208
7.4. Conclusion … 208
7.5. Bibliography … 209
PART 3: FUNCTIONS … 215
Chapter 8. The Role of Emotions in Human-Machine Interaction … 217
Valérie MAFFIOLO and Magalie OCHS
8.1. Introduction … 217
8.2. Interactive information and assistance systems … 219
8.2.1. Uses of emotions in interactive systems … 220
8.2.2. Current research and tools … 220
8.2.2.1. Collecting and annotating expressions of emotions … 221
8.2.2.2. Automatic emotion recognition … 222
8.2.2.3. Interface expressivity … 223
8.3. Video games … 227
8.3.1. The importance of emotions in video games … 228
8.3.1.1. Emotions in gaming systems … 228
8.3.1.2. Player emotions … 229
8.3.2. Current research and tools … 231
8.3.2.1. Towards believable emotional virtual characters … 231
8.3.2.2. Towards games that account for player emotions … 233
8.4. Intelligent tutoring systems (ITS) … 235
8.4.1. The importance of emotions in ITS … 235
8.4.1.1. The learner’s emotions … 235
8.4.1.2. Emotions in ITS … 235
8.4.2. Current research and tools … 235
8.5. Discussion and research perspectives … 237
8.6. Bibliography … 238
Chapter 9. Music and Emotions … 247
Donald GLOWINSKI and Antonio CAMURRI
9.1. The growing importance of music in society … 247
9.2. Recognizing emotions and structural characteristics in music … 249
9.2.1. Understanding listeners’ emotional reactions … 249
9.2.2. A categorical or dimensional approach? … 250
9.3. Rules for modeling musical expression of emotions … 251
9.4. Towards a continuous measure of emotional reactions to music … 252
9.5. Multimodality in musical experience … 253
9.5.1. A multimodal research platform for musical expression … 255
9.5.1.1. Level 1 … 257
9.5.1.2. Level 2 … 258
9.5.1.3. Level 3 … 259
9.5.1.4. Level 4 … 259
9.5.1.5. Overview … 259
9.6. Multimodal emotional synthesis in a musical context … 260
9.7. The social active listening paradigm: the collective aspect of emotion … 262
9.7.1. Example: Mappe per Affetti Erranti … 262
9.8. Conclusion and perspectives … 263
9.9. Bibliography … 263
Chapter 10. Literary Feelings in Interactive Fiction … 271
Marc CAVAZZA and David PIZZI
10.1. Introduction: emotions and feelings … 271
10.2. French novels and the representation of feelings … 273
10.3. Madame Bovary: plot and scenes … 275
10.4. Interactive fiction and emotional planning … 280
10.5. Linguistic interaction and emotions … 284
10.6. Emma Bovary’s virtuality … 290
10.7. Conclusion … 294
10.8. Bibliography … 295
Chapter 11. The Design of Emotions: How the Digital is Making Us More Emotional … 299
Annie GENTÈS
11.1. Representing, interpreting and evoking emotions … 299
11.2. Emotion, mimicry and technical devices … 301
11.2.1. Representing emotions and catharsis … 301
11.3. Devices as an alternate source of emotion: photography … 301
11.4. Art and computers: formal beginnings … 303
11.5. The human behind the mechanics and the mechanics behind the human … 305
11.6. Mirror interaction as an emotional vehicle … 307
11.7. Trompe l’œil versus explicit expression … 309
11.8. Three-dimensional universes: an empathetic experience … 311
11.9. Empathy and identifying emotions … 315
11.9.1. From empathy to shared emotions … 315
11.10. Making human?machine interaction and dialog effective … 317
11.11. Conclusion: “revenge of the emotions” … 318
11.12. Bibliography … 318
List of Authors … 321
Index … 325