探索人类评估者群体对面向聊天对话评估的影响
Exploring the Impact of Human Evaluator Group on Chat-Oriented Dialogue Evaluation
AI总结：
本文通过4个评估者群体测试4个最先进对话系统，分析评估者群体对对话评估的影响，发现Likert评估比Pairwise评估对评估者群体更稳健，并揭示了专业水平与客观性对评估结果的影响。
摘要：
人类评估已被广泛接受为评估面向聊天对话系统的标准。然而，以往工作在招募谁作为评估者方面存在显著差异。领域专家、大学生和专业标注员等评估者群体已被用于评估和比较对话系统，但尚不清楚评估者群体的选择会在多大程度上影响结果。本文通过使用4个不同的评估者群体测试4个最先进的对话系统，分析评估者群体对对话系统评估的影响。我们的分析揭示，在Likert评估中对评估者群体具有稳健性，而这种稳健性在Pairwise评估中并未出现，当更换评估者群体时仅观察到微小差异。此外，观察到这种稳健性存在两个显著局限，揭示了具有不同聊天机器人专业水平的评估者之间的差异，并表明评估者的客观性对某些对话指标是有益的。
英文摘要：
Human evaluation has been widely accepted as the standard for evaluating chat-oriented dialogue systems. However, there is a significant variation in previous work regarding who gets recruited as evaluators. Evaluator groups such as domain experts, university students, and professional annotators have been used to assess and compare dialogue systems, although it is unclear to what extent the choice of an evaluator group can affect results. This paper analyzes the evaluator group impact on dialogue system evaluation by testing 4 state-of-the-art dialogue systems using 4 distinct evaluator groups. Our analysis reveals a robustness towards evaluator groups for Likert evaluations that is not seen for Pairwise, with only minor differences observed when changing evaluator groups. Furthermore, two notable limitations to this robustness are observed, which reveal discrepancies between evaluators with different levels of chatbot expertise and indicate that evaluator objectivity is beneficial for certain dialogue metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
