不要忘记你的ABC：评估面向聊天对话系统的最新技术
Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems
AI总结：
本文提出了一种新颖的人工评估方法，用于估计多种对话系统行为的发生率，对四个最先进的开放域对话系统进行维度评估，证明其比Likert风格或比较方法更合适。
摘要：
尽管对话系统取得了巨大进步，但稳定的评估仍需依赖人工判断，由于其固有的主观性，这产生了出了名的高方差指标。此外，对话评估中的方法和标签尚未完全标准化，特别是对于开放域聊天，缺乏对这些方法的有效性进行比较和评估的研究。不一致的评估使用会误导对话系统的性能，这成为提升系统的主要障碍。因此，需要一种对面向聊天的开放域对话系统进行维度评估的方法，以可靠地衡量对话能力的多个方面。本文提出了一种新颖的人工评估方法，用于估计多种对话系统行为的发生率。我们的方法被用于评估四个最先进的开放域对话系统，并与现有方法进行了比较。分析表明，对于这些系统的维度评估，我们的行为方法比替代的 Likert 风格或比较方法更合适。
英文摘要：
Despite tremendous advancements in dialogue systems, stable evaluation still requires human judgments producing notoriously high-variance metrics due to their inherent subjectivity. Moreover, methods and labels in dialogue evaluation are not fully standardized, especially for open-domain chats, with a lack of work to compare and assess the validity of those approaches. The use of inconsistent evaluation can misinform the performance of a dialogue system, which becomes a major hurdle to enhance it. Thus, a dimensional evaluation of chat-oriented open-domain dialogue systems that reliably measures several aspects of dialogue capabilities is desired. This paper presents a novel human evaluation method to estimate the rates of many dialogue system behaviors. Our method is used to evaluate four state-of-the-art open-domain dialogue systems and compared with existing approaches. The analysis demonstrates that our behavior method is more suitable than alternative Likert-style or comparative approaches for dimensional evaluation of these systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
