arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.03706cs.CLcs.AI
收藏

Dialog评估工具的综合评估

A Comprehensive Assessment of Dialog Evaluation Metrics

Carnegie Mellon University(卡内基梅隆大学)

Yi-Ting Yeh, Maxine Eskenazi, Shikib Mehri

摘要:

自动评价指标是对话系统研究的重要组成部分。已知标准语言评估度量对于评估对话是无效的。因此,最近的研究提出了一些新的,对话特定的指标,更好地与人类的判断。由于研究的快速发展,许多指标已经在不同的数据集上进行了评估,目前还没有时间对它们进行系统的比较。为此,本文提供了一个全面的评估最近提出的对话评价指标的一些数据集。在本文中,23个不同的自动评估指标进行了评估,对10个不同的数据集。此外,在不同的环境中评估这些指标,以更好地确定其各自的长处和短处。(1)在回合级别和对话级别上,(2)针对不同的对话长度,(3)针对不同的对话质量(例如,一致性,参与性),(4)对于不同类型的响应生成模型(即,生成、检索、简单模型和最新模型),(5)考虑不同度量的相似性,以及(6)探索不同度量的组合。这一全面的评估提供了几个关于对话评估指标的要点。它还就如何最好地评估评价指标提出了建议,并为今后的工作指明了有希望的方向。

英文摘要:

Automatic evaluation metrics are a crucial component of dialog systems research. Standard language evaluation metrics are known to be ineffective for evaluating dialog. As such, recent research has proposed a number of novel, dialog-specific metrics that correlate better with human judgements. Due to the fast pace of research, many of these metrics have been assessed on different datasets and there has as yet been no time for a systematic comparison between them. To this end, this paper provides a comprehensive assessment of recently proposed dialog evaluation metrics on a number of datasets. In this paper, 23 different automatic evaluation metrics are evaluated on 10 different datasets. Furthermore, the metrics are assessed in different settings, to better qualify their respective strengths and weaknesses. Metrics are assessed (1) on both the turn level and the dialog level, (2) for different dialog lengths, (3) for different dialog qualities (e.g., coherence, engaging), (4) for different types of response generation models (i.e., generative, retrieval, simple models and state-of-the-art models), (5) taking into account the similarity of different metrics and (6) exploring combinations of different metrics. This comprehensive assessment offers several takeaways pertaining to dialog evaluation metrics in general. It also suggests how to best assess evaluation metrics and indicates promising directions for future work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑