REAM $\sharp$:一种用于开放域对话生成的基于引用的评估树的增强方法
REAM$\sharp$: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation
摘要:
缺乏可靠的自动评估指标是开发开放域对话系统的主要障碍。已经提出了各种基于参考的度量来计算预测的响应和一小组参考之间的分数。然而,这些指标与人类判断的相关性并不令人满意。对于基于参考的度量,其可靠性主要取决于两个因素:其测量预测响应和参考响应之间的相似性的能力,以及给定参考集的可靠性。然而,关于后者的讨论很少。我们的工作试图填补这一空缺。我们首先澄清一个假设,基于引用的指标,如果更多的高质量的参考被添加到参考集,度量的可靠性将增加。接下来,我们提出了REAM $\sharp$:一个增强的方法,以参考为基础的评估开放域对话系统。设计了一个预测模型来估计给定参考集的可靠性。我们展示了它的预测结果如何有助于增加参考集,从而提高度量的可靠性。实验验证了我们的预测模型的有效性和基于参考的指标的可靠性提高与增强的参考集。
英文摘要:
The lack of reliable automatic evaluation metrics is a major impediment to the development of open-domain dialogue systems. Various reference-based metrics have been proposed to calculate a score between a predicted response and a small set of references. However, these metrics show unsatisfactory correlations with human judgments. For a reference-based metric, its reliability mainly depends on two factors: its ability to measure the similarity between the predicted response and the reference response, as well as the reliability of the given reference set. Yet, there are few discussions on the latter. Our work attempts to fill this vacancy. We first clarify an assumption on reference-based metrics that, if more high-quality references are added into the reference set, the reliability of the metric will increase. Next, we present REAM$\sharp$: an enhancement approach to Reference-based EvAluation Metrics for open-domain dialogue systems. A prediction model is designed to estimate the reliability of the given reference set. We show how its predicted results can be helpful to augment the reference set, and thus improve the reliability of the metric. Experiments validate both the effectiveness of our prediction model and that the reliability of reference-based metrics improves with the augmented reference sets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。