Natural Language Generation in Interactive Systems (2014) .. edited by Amanda Stent, etc
Contents
List of contributors page xiii
1 Introduction 1
Amanda Stent and Srinivas Bangalore
1.1 Natural language generation 1
1.2 Interactive systems 2
1.3 Natural language generation for interactive systems 2
1.3.1 Collaboration 3
1.3.2 Reference 5
1.3.3 Handling uncertainty 5
1.3.4 Engagement 6
1.3.5 Evaluation and shared tasks 7
1.4 Summary 8
References 8
Part I Joint construction 11
2 Communicative intentions and natural language generation 13
Nate Blaylock
2.1 Introduction 13
2.2 What are communicative intentions? 13
2.3 Communicative intentions in interactive systems 15
2.3.1 Fixed-task models 16
2.3.2 Plan-based models 17
2.3.3 Conversation Acts Theory 21
2.3.4 Rational behavior models 22
2.4 Modeling communicative intentions with problem solving 23
2.4.1 Collaborative problem solving 23
2.4.2 Collaborative problem solving state 24
2.4.3 Grounding 25
2.4.4 Communicative intentions 26
2.5 Implications of collaborative problem solving for NLG 27
2.6 Conclusions and future work 30
References 31
3 Pursuing and demonstrating understanding in dialogue 34
David DeVault and Matthew Stone
3.1 Introduction 34
3.2 Background 35
3.2.1 Grounding behaviors 36
3.2.2 Grounding as a collaborative process 38
3.2.3 Grounding as problem solving 40
3.3 An NLG model for flexible grounding 42
3.3.1 Utterances and contributions 44
3.3.2 Modeling uncertainty in interpretation 47
3.3.3 Generating under uncertainty 47
3.3.4 Examples 49
3.4 Alternative approaches 51
3.4.1 Incremental common ground 52
3.4.2 Probabilistic inference 52
3.4.3 Correlating conversational success with grounding features 53
3.5 Future challenges 54
3.5.1 Explicit multimodal grounding 54
3.5.2 Implicit multimodal grounding 55
3.5.3 Grounding through task action 56
3.6 Conclusions 57
References 58
4 Dialogue and compound contributions 63
Matthew Purver, Julian Hough, and Eleni Gregoromichelaki
4.1 Introduction 63
4.2 Compound contributions 64
4.2.1 Introduction 64
4.2.2 Data 64
4.2.3 Incremental interpretation vs. incremental representation 68
4.2.4 CCs and intentions 69
4.2.5 CCs and coordination 69
4.2.6 Implications for NLG 70
4.3 Previous work 70
4.3.1 Psycholinguistic research 70
4.3.2 Incrementality in NLG 71
4.3.3 Interleaving parsing and generation 72
4.3.4 Incremental NLG for dialogue 72
4.3.5 Computational and formal approaches 73
4.3.6 Summary 75
4.4 Dynamic Syntax (DS) and Type Theory with Records (TTR) 75
4.4.1 Dynamic Syntax 75
4.4.2 Meeting the criteria 79
4.5 Generating compound contributions 82
4.5.1 The DyLan dialogue system 82
4.5.2 Parsing and generation co-constructing a shared data structure 83
4.5.3 Speaker transition points 84
4.6 Conclusions and implications for NLG systems 86
References 88
Part II Reference 93
5 Referability 95
Kees van Deemter
5.1 Introduction 95
5.2 An algorithm for generating boolean referring expressions 97
5.3 Adding proper names to REG 101
5.4 Knowledge representation 103
5.4.1 Relational descriptions 103
5.4.2 Knowledge representation and REG 104
5.4.3 Description Logic for REG 106
5.5 Referability 113
5.6 Why study highly expressive REG algorithms? 119
5.6.1 Sometimes the referent could not be identified before 119
5.6.2 Sometimes they generate simpler referring expressions 119
5.6.3 Simplicity is not everything 120
5.6.4 Complex content does not always require a complex form 120
5.6.5 Characterizing linguistic competence 121
5.7 Whither REG? 122
References 122
6 Referring expression generation in interaction: A graph-based perspective 126
Emiel Krahmer, Martijn Goudbeek, and Mariet Theune
6.1 Introduction 126
6.1.1 Referring expression generation 127
6.1.2 Preferences versus adaptation in reference 127
6.2 Graph-based referring expression generation 129
6.2.1 Scene graphs 129
6.2.2 Referring graphs 129
6.2.3 Formalizing reference in terms of subgraph isomorphism 130
6.2.4 Cost functions 131
6.2.5 Algorithm 131
6.2.6 Discussion 132
6.3 Determining preferences and computing costs 134
6.4 Adaptation and interaction 137
6.4.1 Experiment I: adaptation and attribute selection 138
6.4.2 Experiment II: adaptation and overspecification 141
6.5 General discussion 143
6.6 Conclusion 145
References 146
Part III Handling uncertainty 149
7 Reinforcement learning approaches to natural language generation in interactive systems 151
Oliver Lemon, Srinivasan Janarthanam, and Verena Rieser
7.1 Motivation 151
7.1.1 Background: Reinforcement learning approaches to NLG 152
7.1.2 Previous work in adaptive NLG 153
7.2 Adaptive information presentation 155
7.2.1 Corpus 157
7.2.2 User simulations for training NLG 157
7.2.3 Data-driven reward function 159
7.2.4 Reinforcement learning experiments 159
7.2.5 Results: Simulated users 161
7.2.6 Results: Real users 162
7.3 Adapting to unknown users in referring expression generation 163
7.3.1 Corpus 164
7.3.2 Dialogue manager and generation modules 164
7.3.3 Referring expression generation module 165
7.3.4 User simulations 166
7.3.5 Training the referring expression generation module 167
7.3.6 Evaluation with real users 169
7.4 Adaptive temporal referring expressions 171
7.4.1 Corpus 171
7.4.2 User simulation 171
7.4.3 Evaluation with real users 172
7.5 Research directions 173
7.6 Conclusions 173
References 175
8 A joint learning approach for situated language generation 180
Nina Dethlefs and Heriberto Cuayahuitl
8.1 Introduction 180
8.2 GIVE 181
8.2.1 The GIVE-2 corpus 181
8.2.2 Natural language generation for GIVE 183
8.2.3 Data annotation and baseline NLG system 184
8.3 Hierarchical reinforcement learning for NLG 185
8.3.1 An example 185
8.3.2 Reinforcement learning with a flat state–action space 187
8.3.3 Reinforcement learning with a hierarchical state–action space 187
8.4 Hierarchical reinforcement learning for GIVE 188
8.4.1 Experimental setting 188
8.4.2 Experimental results 192
8.5 Hierarchical reinforcement learning and HMMs for GIVE 193
8.5.1 Hidden Markov models for surface realization 193
8.5.2 Retraining the learning agent 195
8.5.3 Results 196
8.6 Discussion 198
8.7 Conclusions and future work 200
References 201
Part IV Engagement 205
9 Data-driven methods for linguistic style control 207
Francois Mairesse
9.1 Introduction 207
9.2 PERSONAGE: personality-dependent linguistic control 208
9.3 Learning to control a handcrafted generator from data 212
9.3.1 Overgenerate and rank 212
9.3.2 Parameter estimation models 217
9.4 Learning a generator from data using factored language models 220
9.5 Discussion and future challenges 222
References 224
10 Integration of cultural factors into the behavioral models of virtual characters 227
Birgit Endrass and Elisabeth Andre
10.1 Introduction 227
10.2 Culture and communicative behaviors 229
10.2.1 Levels of culture 229
10.2.2 Cultural dichotomies 231
10.2.3 Hofstede’s dimensional model and synthetic cultures 232
10.3 Levels of cultural adaptation 233
10.3.1 Culture-specific adaptation of context 233
10.3.2 Culture-specific adaptation of form 234
10.3.3 Culture-specific communication management 235
10.4 Approaches to culture-specific modeling for embodied virtual agents 236
10.4.1 Top-down approaches 236
10.4.2 Bottom-up approaches 237
10.5 A hybrid approach to integrating culture-specific behaviors into virtual agents 238
10.5.1 Cultural profiles for Germany and Japan 239
10.5.2 Behavioral expectations for Germany and Japan 239
10.5.3 Formalization of culture-specific behavioral differences 242
10.5.4 Computational models for culture-specific conversational behaviors 245
10.5.5 Simulation 246
10.5.6 Evaluation 247
10.6 Conclusions 248
References 248
11 Natural language generation for augmentative and assistive technologies 252
Nava Tintarev, Ehud Reiter, Rolf Black, and Annalu Waller
11.1 Introduction 252
11.2 Background on augmentative and alternative communication 253
11.2.1 State of the art 253
11.2.2 Related research 256
11.2.3 Diversity in users of AAC 257
11.2.4 Other AAC challenges 258
11.3 Application areas of NLG in AAC 259
11.3.1 Helping AAC users communicate 259
11.3.2 Teaching communication skills to AAC users 260
11.3.3 Accessibility: Helping people with visual impairments access information 260
11.3.4 Summary 261
11.4 Example project: “How was School Today…?” 261
11.4.1 Use case 261
11.4.2 Example interaction 262
11.4.3 NLG in “How was School Today…?” 262
11.4.4 Current work on “How was School Today…?” 268
11.5 Challenges for NLG and AAC 272
11.5.1 Supporting social interaction 272
11.5.2 Narrative 273
11.5.3 User personalization 273
11.5.4 System evaluation 273
11.5.5 Interaction and dialogue 274
11.6 Conclusions 274
References 274
Part V Evaluation and shared tasks 279
12 Eye tracking for the online evaluation of prosody in speech synthesis 281
Michael White, Rajakrishnan Rajkumar, Kiwako Ito, and Shari R. Speer
12.1 Introduction 281
12.2 Experiment 283
12.2.1 Design and materials 283
12.2.2 Participants and eye-tracking procedure 285
12.3 Results 286
12.4 Interim discussion 289
12.5 Offline ratings 290
12.5.1 Design and materials 290
12.5.2 Results 291
12.6 Acoustic analysis using Generalized Linear Mixed Models (GLMMs) 292
12.6.1 Acoustic factors and looks to the area of interest 293
12.6.2 Relationship between ratings and looks 295
12.6.3 Correlation between rating and acoustic factors 296
12.7 Discussion 297
12.8 Conclusions 298
References 299
13 Comparative evaluation and shared tasks for NLG in interactive systems 302
Anja Belz and Helen Hastie
13.1 Introduction 302
13.2 A categorization framework for evaluations of automatically generated language 304
13.2.1 Evaluation measures 304
13.2.2 Higher-level quality criteria 307
13.2.3 Evaluation frameworks 308
13.2.4 Concluding comments 309
13.3 An overview of evaluation and shared tasks in NLG 309
13.3.1 Component evaluation: Referring Expression Generation 309
13.3.2 Component evaluation: Surface Realization 315
13.3.3 End-to-end NLG systems: data-to-text generation 316
13.3.4 End-to-end NLG systems: text-to-text generation 317
13.3.5 Embedded NLG components 319
13.3.6 Embedded NLG components: the GIVE shared task 322
13.3.7 Concluding comments 325
13.4 An overview of evaluation for spoken dialogue systems 325
13.4.1 Introduction 325
13.4.2 Realism and control 325
13.4.3 Evaluation frameworks 327
13.4.4 Shared tasks 329
13.4.5 Discussion 331
13.4.6 Concluding comments 331
13.5 A methodology for comparative evaluation of NLG components in interactive systems 332
13.5.1 Evaluation model design 332
13.5.2 An evaluation model for comparative evaluation of NLG modules in interactive systems 333
13.5.3 Context-independent intrinsic output quality 334
13.5.4 Context-dependent intrinsic output quality 335
13.5.5 User satisfaction 336
13.5.6 Task effectiveness and efficiency 336
13.5.7 System purpose success 336
13.5.8 A proposal for a shared task on referring expression generation in dialogue context 337
13.5.9 GRUVE: A shared task on instruction giving in pedestrian navigation 339
13.5.10 Concluding comments 341
13.6 Conclusion 341
References 341
Author index 351
Subject index 359