arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.04720cs.CL
收藏

DEnsity:基于密度估计的开放域对话评估指标

DEnsity: Open-domain Dialogue Evaluation Metric using Density Estimation

KAIST AI(韩国科学技术院人工智能学院)

ChaeHun Park, Seungil Chad Lee, Daniel Rim, Jaegul Choo

AI总结:

针对开放域对话评估中分类器过度自信的问题,本文提出基于密度估计的指标DEnsity,并结合对比学习压缩特征空间,在多个数据集上取得优于现有指标的人类评估相关性。

摘要:

尽管开放域对话系统近期取得了进展,构建可靠的评估指标仍然是一个具有挑战性的问题。近期研究提出了基于分类模型的可学习指标,这些模型经过训练以区分正确回复。然而,已知神经分类器会对来自未见分布的样本做出过度自信的预测。我们提出 DEnsity,它利用从神经分类器导出的特征空间上的密度估计来评估回复。我们的指标衡量一个回复在人类对话分布中出现的可能性。此外,为了提高 DEnsity 的性能,我们利用对比学习进一步压缩特征空间。在多个回复评估数据集上的实验表明,DEnsity 与人类评估的相关性优于现有指标。我们的代码可在 https://github.com/ddehun/DEnsity 获取。

英文摘要:

Despite the recent advances in open-domain dialogue systems, building a reliable evaluation metric is still a challenging problem. Recent studies proposed learnable metrics based on classification models trained to distinguish the correct response. However, neural classifiers are known to make overly confident predictions for examples from unseen distributions. We propose DEnsity, which evaluates a response by utilizing density estimation on the feature space derived from a neural classifier. Our metric measures how likely a response would appear in the distribution of human conversations. Moreover, to improve the performance of DEnsity, we utilize contrastive learning to further compress the feature space. Experiments on multiple response evaluation datasets show that DEnsity correlates better with human evaluations than the existing metrics. Our code is available at https://github.com/ddehun/DEnsity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑