Achieving Reliable Human Assessment of Open-Domain Dialogue Systems
ADAPT Centre（ADAPT中心）；Dublin City University（都柏林城市大学）；Trinity College Dublin（都柏林圣三一学院）；SFI Centre for Research Training in Machine Learning（爱尔兰科学基金会机器学习研究培训中心）；Huawei（华为）；Noah’s Ark Lab（诺亚方舟实验室）
Tianbo Ji, Yvette Graham, Gareth J. F. Jones, Chenyang Lyu, Qun Liu
Evaluation of open-domain dialogue systems is highly challenging and development of better techniques is highlighted time and again as desperately needed. Despite substantial efforts to carry out reliable live evaluation of systems in recent competitions, annotations have been abandoned and reported as too unreliable to yield sensible results. This is a serious problem since automatic metrics are not known to provide a good indication of what may or may not be a high-quality conversation. Answering the distress call of competitions that have emphasized the urgent need for better evaluation techniques in dialogue, we present the successful development of human evaluation that is highly reliable while still remaining feasible and low cost. Self-replication experiments reveal almost perfectly repeatable results with a correlation of $r=0.969$. Furthermore, due to the lack of appropriate methods of statistical significance testing, the likelihood of potential improvements to systems occurring due to chance is rarely taken into account in dialogue evaluation, and the evaluation we propose facilitates application of standard tests. Since we have developed a highly reliable evaluation method, new insights into system performance can be revealed. We therefore include a comparison of state-of-the-art models (i) with and without personas, to measure the contribution of personas to conversation quality, as well as (ii) prescribed versus freely chosen topics. Interestingly with respect to personas, results indicate that personas do not positively contribute to conversation quality as expected.
PoE: a Panel of Experts for Generalized Automatic Dialogue Assessment
National University of Singapore（新加坡国立大学）；Universidad Politécnica de Madrid（马德里理工大学）；Robert Bosch (SEA) Pte Ltd（博世（东南亚）私人有限公司）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation
National University of Singapore（新加坡国立大学）；Robert Bosch (SEA)（罗伯特·博世（东南亚）公司）；Kriston AI Lab（克里斯顿人工智能实验室）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））
Learning to Plan and Realize Separately for Open-Ended Dialogue Systems
University of North Carolina Charlotte（北卡罗来纳大学夏洛特分校）；Institute for Human Machine Cognition (IHMC)（人机认知研究所）；State University of New York Albany（纽约州立大学奥尔巴尼分校）；Rensselaer Polytechnic Institute（伦斯勒理工学院）
FCM: A Fine-grained Comparison Model for Multi-turn Dialogue Reasoning
Beijing University of Posts and Telecommunications（北京邮电大学）；Data Science Lab, JD.com（京东数据科学实验室）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）
DiSCoL: Toward Engaging Dialogue Systems through Conversational Line Guided Response Generation
Information Sciences Institute, University of Southern California（南加州大学信息科学研究所）；Columbia University（哥伦比亚大学）；University of California Los Angeles（加州大学洛杉矶分校）
An Efficient Task-Oriented Dialogue Policy: Evolutionary Reinforcement Learning Injected by Elite Individuals
School of Computer Science and Technology, Changsha University of Science and Technology, China（长沙理工大学计算机科学与技术学院）；School of Computer Science and Engineering, Central South University, China（中南大学计算机科学与工程学院）；School of Information and Computing Sciences, Utrecht University, Netherlands（乌特勒支大学信息与计算科学学院）
Unsupervised Flow Discovery from Task-oriented Dialogues
CISUC, Universidade de Coimbra（科英布拉大学系统与计算机工程中心，科英布拉大学）；DEI, FCTUC, Universidade de Coimbra（科英布拉大学工程学院，科英布拉大学）；ISEC, Instituto Politécnico de Coimbra（科英布拉理工学院高等工程学院）
Divide, Conquer, and Combine: Mixture of Semantic-Independent Experts for Zero-Shot Dialogue State Tracking
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；JD Explore Academy, JD.com Inc（京东集团京东探索研究院）；The University of Sydney（悉尼大学）
Dial-MAE: ConTextual Masked Auto-Encoder for Retrieval-based Dialogue Systems
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple LLM Judges
ZJU–UIUC Institute（浙大UIUC研究院）；ZJU–Hangzhou Global Scientific and Technological Innovation Center（浙大杭州全球科技创新中心）；College of Computer Science and Technology（计算机科学与技术学院）；Alibaba Group（阿里巴巴集团）
Systematic Exploration of Dialogue Summarization Approaches for Reproducibility, Comparative Assessment, and Methodological Innovations for Advancing Natural Language Processing in Abstractive Summarization
SRM Institute of Science and Technology（SRM科学技术学院）；SRM University, AP（SRM大学（安得拉邦））
当前的对话系统对于用户来说不是很吸引人，特别是当在不依赖于主动重新参与脚本策略的情况下进行端到端培训时。Zhang et al.（2018）表明，当将端到端对话模型的参与度调节为文本角色时，会增加模型的参与度，为模型提供一些个性化的背景故事。然而，Zhang et al.（2018）中使用的数据集是合成的，大小有限，因为它包含大约1k个不同的人物角色。在本文中，我们介绍了一个新的数据集，提供500万个人物角色和7亿个基于人物角色的对话。我们的实验表明，在这种规模下，使用人物角色进行训练仍然可以提高端到端系统的性能。此外，我们还通过对Zhang et al.（2018）的数据微调我们的模型并实现最先进的结果，证明了其他任务也受益于我们数据集的广泛覆盖。
