Interactive Multi-modal Question-answering


Interactive Multi-modal Question-answering (2011) Antal van den Bosch @antalvdb & Gosse Bouma @gosseb eds.


Contents

Part I Introduction to the IMIX Programme

Introduction . . . 3

Antal van den Bosch and Gosse Bouma

1 Interactive Multimodal Question Answering . . . 3

2 The IMIX Project . . . 4

3 Contributions . . . 5

4 Further Reading . . . 7

References . . . 8

The IMIX Demonstrator: An Information Search Assistant for the Medical Domain . . . 11

1 Introduction . . . 11

2 A Medical Information Search Assistant . . . 12

3 Architecture of the Final Version . . . 14

3.1 The Modules . . . 16

3.2 The DAM State Machine . . . 18

3.3 A Modular Version of the Demonstrator . . . 18

4 Conclusion . . . 19

References . . . 21

Part II Interaction Management

Vidiam: Corpus-based Development of a Dialogue Manager for Multimodal Question Answering . . . 25

Boris van Schooten and Rieks op den Akker

1 Introduction . . . 25

1.1 QA Dialogue System Features . . . 26

2 Overview of Existing Systems . . . 29

2.1 FQ Context Completion Strategies . . . 30

3 The Corpora . . . 35

3.1 The Follow-up Question Corpus . . . 35

3.2 The Multimodal Follow-up Question Corpus . . . 38

3.3 The Ritel Corpus . . . 42

4 The Dialogue Manager . . . 45

4.1 FU Classification Performance . . . 45

4.2 Rewriting and Context Completion Performance . . . 47

4.3 Answering Performance . . . 50

5 Conclusions . . . 53

References . . . 54

Multidimensional Dialogue Management. . . 57

1 Introduction . . . 57

2 Semantic and Pragmatic Framework: DIT . . . 59

2.1 Dimensions and Communicative Functions . . . 59

3 Multifunctionality . . . 63

3.1 Relations Between Communicative Functions . . . 63

3.2 Types of Multifunctionality in Dialogue Units . . . 66

4 Design of a Multidimensional Dialogue Manager. . . 69

4.1 Context Model . . . 69

4.2 Dialogue Act Agents . . . 70

4.3 Application: Dialogue Management for Interactive QA . . 72

5 Context Specification and Update Mechanisms. . . 74

5.1 Specification of the Context Model . . . 75

5.2 Levels of Processing and Feedback . . . 75

5.3 Grounding . . . 76

5.4 Context Update Model . . . 77

6 Constraints on Generating Combinations of Dialogue Acts . . . 78

6.1 Logical Constraints . . . 78

6.2 Pragmatic Constraints . . . 79

6.3 Constraints for Segment Sequences . . . 80

6.4 Constraints Defining Dialogue Strategies . . . 80

6.5 Evaluation Agent Design . . . 83

7 Conclusion . . . 84

References . . . 85

Part III Fusing Text, Speech, and Images

Experiments in Multimodal Information Presentation . . . 89

Mariet Theune

1 Introduction . . . 89

2 Experiment 1: Production of Multimodal Answers . . . 91

2.1 Participants . . . 92

2.2 Stimuli . . . 92

2.3 Coding System and Procedure . . . 92

2.4 Results . . . 93

2.5 Conclusion . . . 96

3 Experiment 2: Evaluation of Multimodal Answers . . . 96

3.1 Participants . . . 96

3.2 Design. . . 97

3.3 Stimuli . . . 97

3.4 Procedure . . . 98

3.5 Results . . . 98

3.6 Conclusion . . . 101

4 Automatic Production of Multimodal Answers . . . 102

4.1 Multimedia Summarization . . . 102

4.2 Automatic Picture Selection . . . 103

5 Experiment 3: Evaluating Automatically Produced Multimodal Answers . . . 104

5.1 Participants . . . 105

5.2 Design. . . 105

5.3 Stimuli . . . 106

5.4 Procedure . . . 107

5.5 Data Processing . . . 108

5.6 Results . . . 108

5.7 Conclusion . . . 112

6 General Discussion . . . 112

References . . . 114

Text-to-Text Generation for Question Answering . . . 117

Wauter Bosma, Erwin Marsi, Emiel Krahmer and Mariet Theune

1 Introduction . . . 117

2 Graph-based Content Selection . . . 119

2.1 Related Work . . . 119

2.2 Task Definition . . . 121

2.3 A Framework for Summarisation . . . 122

2.4 Query-based Summarisation . . . 122

2.5 Results . . . 129

2.6 Validating the Results . . . 130

3 Sentence Fusion . . . 131

3.1 Data Collection and Annotation. . . 133

3.2 Automatic Alignment . . . 137

3.3 Merging and Generation . . . 139

3.4 Discussion . . . 141

4 Conclusion . . . 142

References . . . 143

Part IV Text Analysis for Question Answering

Automatic Extraction of Medical Term Variants from Multilingual Parallel Translations . . . 149

1 Introduction . . . 149

2 Alignment-based Methods . . . 153

2.1 Translational Context . . . 153

2.2 Measures for Computing Semantic Similarity . . . 155

2.3 Related Work . . . 156

3 Materials and Methods . . . 158

3.1 The multilingual Parallel Corpus EMEA . . . 159

3.2 Automatic Word Alignment and Phrase Extraction . . . 159

3.3 Selecting Candidate Terms . . . 160

3.4 Comparing Translation Vectors . . . 161

3.5 Post-processing . . . 162

4 Evaluation . . . 162

4.1 Gold Standard . . . 163

4.2 Test Set . . . 163

5 Results and Discussion . . . 163

5.1 Two Methods for Comparison . . . 163

5.2 Results . . . 164

5.3 Error Analysis . . . 166

6 Conclusions . . . 167

References . . . 168

Relation Extraction for Open and Closed Domain Question Answering . . 171

1 Introduction . . . 172

2 Related Work . . . 174

2.1 Relation Extraction for Open Domain QA . . . 174

2.2 Biomedical Relation Extraction . . . 175

2.3 Using Syntactic Patterns . . . 175

3 Dependency Information for Question Answering and Relation Extraction . . . 177

4 Relation Extraction for Open Domain QA. . . 179

4.1 Pattern Induction . . . 180

4.2 Experiment . . . 181

4.3 Evaluation . . . 183

5 Relation Extraction for Medical QA. . . 186

5.1 Multilingual Term Labelling . . . 187

5.2 Learning Patterns . . . 189

5.3 Evaluation . . . 190

5.4 Evaluation in a QA Setting . . . 192

6 Conclusions and Future Work . . . 193

References . . . 195

Constraint-Satisfaction Inference for Entity Recognition . . . 199

1 Introduction . . . 199

2 Sequence Labelling . . . 200

3 Related Work . . . 201

4 A Baseline Approach . . . 202

4.1 Class Trigrams . . . 203

4.2 Memory-based Learning . . . 204

5 Constraint Satisfaction Inference . . . 205

5.1 Solving the CSP . . . 208

6 Sequence Labelling Tasks . . . 208

6.1 Syntactic Chunking . . . 209

6.2 Named-Entity Recognition . . . 210

6.3 Medical Concept Chunking: The IMIX Task . . . 211

7 Experimental Set-up . . . 212

7.1 Evaluation . . . 212

7.2 Constraint Prediction . . . 213

8 Results . . . 214

8.1 Comparison to Alternative Techniques . . . 215

9 Discussion . . . 216

9.1 Other Constraint-based Approaches to Sequence Labelling . . . 217

10 Conclusion . . . 218

References . . . 219

Extraction of Hypernymy Information from Text . . . 223

Erik Tjong Kim Sang, Katja Hofmann and Maarten de Rijke

1 Introduction . . . 223

2 Task and Approach . . . 224

2.1 Task . . . 224

2.2 Natural Language Processing . . . 225

2.3 Collecting Evidence . . . 226

2.4 Evaluation . . . 228

3 Study 1: Comparing Pattern Numbers and Corpus Sizes . . . 229

3.1 Extracting Individual Patterns . . . 229

3.2 Combining Corpus Patterns . . . 230

3.3 Web Query Format . . . 232

3.4 Web Extraction Results . . . 233

3.5 Error Analysis . . . 234

3.6 Discussion . . . 234

4 Study 2: Examining the Effect of Ambiguity. . . 235

4.1 Approach . . . 235

4.2 Experiments and Results . . . 235

4.3 Discussion . . . 237

5 Study 3: Measuring the Effect of Syntactic Processing . . . 238

5.1 Experiments and Results . . . 238

5.2 Result Analysis . . . 240

5.3 Discussion . . . 243

6 Concluding Remarks . . . 243

References . . . 244

Towards a Discourse-driven Taxonomic Inference Model . . . 247

Piroska Lendvai

1 Introduction . . . 247

1.1 Conceptual Taxonomy . . . 249

1.2 Discourse Structure . . . 250

1.3 Semantic Inference . . . 251

1.4 Semi-supervised Harvesting of Lexico-semantic Patterns 252

1.5 Related Research in Language Technology . . . 253

2 Exploratory Data . . . 254

2.1 Semantic Annotation Types . . . 254

3 Machine Learning of Taxonomy Identification . . . 256

3.1 Feature Construction . . . 256

3.2 Experimental set-up . . . 256

3.3 Results . . . 257

4 The Taxonomy Inference Model and Textual Entailment . . . 258

5 Extraction of Patterns Involving Medical Concept Types . . . 262

5.1 Masking . . . 262

5.2 Experimental Results . . . 263

6 Closing . . . 265

References . . . 266

Part V Epilogue

IMIX: Good Questions, Promising Answers . . . 271

Eduard Hovy, Jon Oberlander and Norbert Reithinger

1 The Legacy of the IMIX Programme . . . 271

2 Evaluation of the IMIX Programme Work . . . 272

2.1 Technical Evaluation . . . 273

2.2 Programmatic Evaluation . . . 276

2.3 Delivery and Outreach . . . 277

3 Recommendations for the Future . . . 277

References . . . 279