PairEval：基于成对比较的开放域对话评估
PairEval: Open-domain Dialogue Evaluation with Pairwise Comparison
AI总结：
提出PairEval，一种基于成对比较的开放域对话评估指标，利用开源中等规模语言模型实现，在多个基准上比基线更贴近人类判断，并更稳健地检测重复和说话者不敏感等故障。
摘要：
构建可靠且自动化的评估指标对于开放域对话系统而言是必要但具有挑战性的问题。近期研究提出了通过考虑生成响应与先前对话历史的相关性来评估其质量的指标。尽管这些指标有效，但它们直接评估单个响应，而非考虑其相对于其他响应的质量。为解决这一问题，我们提出了PairEval，一种新颖的对话评估指标，通过比较不同对话中响应的质量来评估响应。PairEval基于开源且中等规模的语言模型构建，并使其专门用于对话响应之间的成对比较。在多个基准上的大量实验表明，我们的指标与人类判断的相关性高于基线指标。我们还发现，所提出的比较性指标在检测开放域对话系统的常见故障（包括重复和说话者不敏感）方面更为稳健。
英文摘要：
Building a reliable and automated evaluation metric is a necessary but challenging problem for open-domain dialogue systems. Recent studies proposed evaluation metrics that assess generated responses by considering their relevance to previous dialogue histories. Although effective, these metrics evaluate individual responses directly rather than considering their relative quality compared to other responses. To handle this, we propose PairEval, a novel dialogue evaluation metric for assessing responses by comparing their quality against responses in different conversations. PairEval is built on top of open-sourced and moderate-size language models, and we make them specialized in pairwise comparison between dialogue responses. Extensive experiments on multiple benchmarks demonstrate that our metric exhibits a higher correlation with human judgments than baseline metrics. We also find that the proposed comparative metric is more robust in detecting common failures from open-domain dialogue systems, including repetition and speaker insensitivity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
