Large-scale Hybrid Approach for Predicting User Satisfaction with Conversational Agents
Alexa AI（Alexa人工智能）；Amazon（亚马逊）
Dookun Park, Hao Yuan, Dongmin Kim, Yinglei Zhang, Matsoukas Spyros, Young-Bum Kim, Ruhi Sarikaya, Edward Guo, Yuan Ling, Kevin Quinn, Pham Hung, Benjamin Yao, Sungjin Lee
Measuring user satisfaction level is a challenging task, and a critical component in developing large-scale conversational agent systems serving the needs of real users. An widely used approach to tackle this is to collect human annotation data and use them for evaluation or modeling. Human annotation based approaches are easier to control, but hard to scale. A novel alternative approach is to collect user's direct feedback via a feedback elicitation system embedded to the conversational agent system, and use the collected user feedback to train a machine-learned model for generalization. User feedback is the best proxy for user satisfaction, but is not available for some ineligible intents and certain situations. Thus, these two types of approaches are complementary to each other. In this work, we tackle the user satisfaction assessment problem with a hybrid approach that fuses explicit user feedback, user satisfaction predictions inferred by two machine-learned models, one trained on user feedback data and the other human annotation data. The hybrid approach is based on a waterfall policy, and the experimental results with Amazon Alexa's large-scale datasets show significant improvements in inferring user satisfaction. A detailed hybrid architecture, an in-depth analysis on user feedback data, and an algorithm that generates data sets to properly simulate the live traffic are presented in this paper.
Personalizing Emotion-aware Conversational Agents? Exploring User Traits-driven Conversational Strategies for Enhanced Interaction
KTH Royal Institute of Technology（皇家理工学院）；University of Bergen（卑尔根大学）；University of Surrey（萨里大学）；Norwegian University of Science and Technology（挪威科技大学）；Chalmers University of Technology（查尔姆斯理工大学）
Collaboration with Conversational AI Assistants for UX Evaluation: Questions and How to Ask them (Voice vs. Text)
Rochester Institute of Technology（罗切斯特理工学院）；University of Waterloo（滑铁卢大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）
StorySage: Conversational Autobiography Writing Powered by a Multi-Agent Framework
Department of Management Science and Engineering, Stanford University（管理科学与工程系，斯坦福大学）；Department of Electrical Engineering, Stanford University（电气工程系，斯坦福大学）；Department of Computer Science, Stanford University（计算机科学系，斯坦福大学）
