From Brows to Trust - Evaluating Embodied Conversational Agents

Contents

Contributing Authors

Preface

I EVALUATION METHODOLOGY

Chapter 1 THE BLIND MEN AND THE ELEPHANT REVISITED

1. Introduction

2. The Taxonomy

2.1 Believability

2.2 Sociability

2.3 Task and Application Domains

2.4 Agency and Computational Issues

3. Applying the Taxonomy

4. On Production Values

5. Further Classi.cation

6. Conclusions

Notes

References

Chapter 2 EMBODIED CONVERSATIONAL AGENTS ON A COMMON GROUND

1. Introduction

1.1 Motivations and Problems

2. ECAs from a Design Perspective

2.1 The Embodiment

2.2 The Mental Aspects

2.3 Implementation Aspects

2.4 Range of Applicability

3. On Evaluation Methodology

3.1 Why to Evaluate?

3.2 How to De.ne the Evaluation Variables?

3.3 Testing by what Users?

3.4 How to Collect and Evaluate Data?

4. Dimensions of Evaluation

4.1 Usability

4.2 Evaluation of User Perception of ECAs

5. Conclusions

Notes

References

Chapter 3 EMPIRICAL EVALUATION METHODOLOGY FOR EMBODIED CONVERSATIONAL AGENTS

1. Introduction

2. Types of Evaluation Research

2.1 Star Life Cycle

2.2 Formative and Summative Evaluation

3. Research Model, Research Question and Construct

3.1 Research Model

3.2 Research Questions

3.3 Psychological Construct

4. Research Strategy

4.1 Survey

4.2 Experiment

4.3 Case Study

5. Data Collection Methods

5.1 Qualitative Methods

5.2 Quantitative Methods

5.3 More Data Collection Methods

6. Samples, Reliability and Validity

6.1 Samples

6.2 Reliability

6.3 Validity

7. Data Analysis

8. Concluding: Guidelines for Evaluating ECAs

Acknowledgments

Notes

References

Chapter 4 EVALUATING USERS’ REACTIONS TO HUMAN-LIKE INTERFACES

1. Evaluation of Dialogue Systems

1.1 Evaluation of User’s Satisfaction

1.2 Evaluation Criteria for Multimodal Dialogue Systems

2. Prosodic Cues and Non-verbal Behaviour as new Evaluation Measures

2.1 Prosodic and Visual Cues of Emotions

2.2 Prosodic and Visual Cues of Emotions and Evaluation

3. The Investigation

3.1 Material

3.2 Analysis of Prosodic Cues

3.3 Analysis of Non-verbal Communicative Behaviour

4. Conclusions and Further Investigation

Acknowledgments

Notes

References

II THE USER IN FOCUS

Chapter 5 USER-CENTRED DESIGN AND EVALUATION OF AFFECTIVE INTERFACES

1. Introduction

2. Underlying Philosophy and Method

2.1 Our Philosophy

2.2 Our Method

3. Studies of Three Affective Interfaces

4. Agneta & Frida

4.1 Non-correlation of Measurements

4.2 Narrative Experience

4.3 Implications for Design Method

5. The Influencing Machine

5.1 Study Method

5.2 The First In.uencing Machine Study

5.3 The Second Influencing Machine Study

5.4 Implications for Design Method

6. SenToy and FantasyA

6.1 Wizard of Oz

6.2 Second Study of SenToy Used in FantasyA

6.3 Implications for Design Method

7. Discussion

Acknowledgments

Notes

References

Chapter 6 ‘USER AS ASSESSOR’ APPROACH TO EMBODIED CONVERSATIONAL AGENTS

1. Introduction

2. Attention as an Example of the ‘User as Assessor’ Approach

2.1 How Do Humans Attend?

2.2 Conceptualizing Apparent Attention

3. Creating Apparently Attentive ECAs

3.1 Apparent Selectivity in ECAs

3.2 Apparent Breadth in ECAs

4. Conclusion

Notes

References

III EVALUATION OF ECAS

Chapter 7 MORE ABOUT BROWS

1. Introduction

2. About Brows

3. Materials

3.1 Speech

3.2 Animations

4. Experiment 1: Subjective Preference

4.1 Method

4.2 Results (Dutch)

4.3 Results (Italian)

4.4 Discussion

5. Experiment 2: Perceived Prominence

5.1 Method

5.2 Results (Dutch)

5.3 Results (Italian)

5.4 Discussion

6. Experiment 3: Functional Analysis

6.1 Method

6.2 Results (Dutch)

6.3 Results (Italian)

6.4 Discussion

7. General Discussion

7.1 Eyebrows in Dutch and Italian

7.2 About Analysis-by-Synthesis

7.3 Analysis-by-observation

Acknowledgments

Notes

References

Chapter 8 EVALUATION OF MULTIMODAL BEHAVIOUR OF EMBODIED AGENTS

1. Introduction

2. Experimental Setting

2.1 Participants

2.2 Apparatus

2.3 Scenarios

2.4 Independent Variables

2.5 Generation of Multimodal Behaviour

2.6 Dependent Variables

2.7 Data Analysis

3. Results

3.1 Subjective Variables

3.2 Recall Performance

4. Discussion

4.1 Effects of Multimodal Strategies

4.2 Effects of ECAs’ Appearance

4.3 Additional Results

5. Conclusions and Future Directions

Acknowledgments

Notes

References

Chapter 9 ECA AS USER INTERFACE PARADIGM

1. Introduction

1.1 Our Research Objectives

2. Research Framework

2.1 Features of the User

2.2 Features of the ECA

2.3 Features of the Task

2.4 Interaction of Variables

2.5 Approaches to Assessing ECAs

3. An Experimental Study

3.1 Goal and Hypotheses

3.2 Participants, Materials, and Procedure

3.3 Results

3.4 Additional Observations from the Experiment

4. Conclusions

Notes

References

IV EVALUATION OF APPLICATIONS

Chapter 10 TALKING TO DIGITAL FISH

1. Introduction

1.1 Evaluating Animated Characters in Educational Software

1.2 Goals of the Study

2. Methods

2.1 Participants, Task, and Procedure

2.2 Simulation Environment

2.3 Text to Speech Manipulation

2.4 Research Design and Analyses

2.5 Data Coding and Dependent Measures

3. Results

3.1 Engagement in Interface and Ease of Use

3.2 Distribution of Question Types

3.3 Impact of TTS Voice Type on Child Queries

4. Discussion

4.1 Acoustic Characteristics of Animated Character Design

4.2 Conversational Interfaces as Educational Interfaces

4.3 Conclusion

Acknowledgments

Notes

References

Chapter 11 EXPERIMENTAL EVALUATION OF THE USE OF ECAS IN ECOMMERCE APPLICATIONS

1. Introduction

2. Experimental Approach

3. The Phoebe Experiment

3.1 Experimental Procedure

3.2 The Usability Questionnaire

3.3 Results

3.4 Summary

4. The Voice Personae Experiment

4.1 Voice Personae

4.2 Experimental Procedure

4.3 Results

4.4 Summary

5. The Trust Experiment

5.1 Experimental Procedure

5.2 Results

5.3 Summary

6. Conclusion

References

Chapter 12 WHAT WE CAN LEARN FROM AVATAR-DRIVEN INTERNET COMMUNITIES

1. Introduction

2. The sysis NetLife

2.1 The NetLife Platform

2.2 The Flirtboat Application

2.3 The derSpittelberg Application

3. User Data Analysis

3.1 Goals for Data Collection

3.2 Methodology

3.3 Generalized Avatar Pro.le

3.4 Lifestyle Analysis

4. Data Evaluation

4.1 Evaluation Results

5. Conclusion

Acknowledgments

Notes

References

More eBooks at www.ciando.com