CausalScore：一种用于评估开放域对话系统响应相关性的无参考自动评估指标
CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systems
AI总结：
提出无参考自动评估指标CausalScore，通过测量对话历史与响应间的因果强度评估相关性，实验证明其与人类判断的一致性显著优于现有最先进指标，并构建了含人工标注因果关系的数据集CGDIALOG+。
摘要：
自动评估开放域对话系统中响应的质量是一项具有挑战性但至关重要的任务。当前的评估指标通常无法与人类判断保持一致，特别是在评估语法正确的响应时。为了解决这个问题，我们提出了一种名为CausalScore的新指标，它通过测量对话历史与响应之间的因果强度来评估响应的相关性。因果强度的估计利用了从对话历史到响应的无条件依赖和条件依赖。我们将我们的指标与现有的竞争指标在它们与人类判断的一致性方面进行了比较。我们的实验结果表明，CausalScore通过与人类判断更好地保持一致，显著超越了现有的最先进指标。此外，我们收集了一个新的对话数据集CGDIALOG+，其中包含人工标注的因果关系和一组成对的人类判断，以促进未来自动指标的开发。
英文摘要：
Automatically evaluating the quality of responses in open-domain dialogue systems is a challenging but crucial task. Current evaluation metrics often fail to align with human judgments, especially when assessing responses that are grammatically correct. To address this issue, we propose a novel metric, called CausalScore, which assesses the relevance of responses by measuring the causal strength between dialogue histories and responses. The causal strength is estimated by utilizing both unconditional dependence and conditional dependencies from the dialogue history to responses. We compare our metric with the existing competitive metrics in terms of their alignment with human judgements. Our experimental results demonstrate that CausalScore significantly surpasses existing state-of-the-art metrics by aligning better with human judgements. Additionally, we collect a new dialogue dataset CGDIALOG+ with human-annotated causal relations and a set of pairwise human judgements to facilitate the development of future automatic metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
