arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2006.12719cs.CLcs.AIcs.HC
收藏

基于DialoGPT的交互式对话的无监督评估

Unsupervised Evaluation of Interactive Dialog with DialoGPT

Dialog Research Center, Language Technologies Institute(语言技术研究所对话研究中心);Carnegie Mellon University(卡内基梅隆大学)

Shikib Mehri, Maxine Eskenazi

摘要:

定义有意义和可解释的自动评估指标对于开放域对话研究是非常重要的。标准的语言生成度量已被证明对对话无效。本文介绍了FED度量(对话的细粒度评估),这是一种使用DialoGPT的自动评估度量,无需任何微调或监督。它还介绍了FED数据集,该数据集是通过注释一组具有18个细粒度对话质量的人-系统和人-人对话而构建的。FED度量(1)不依赖于地面实况响应,(2)不需要训练数据,以及(3)在回合和整个对话级别上测量细粒度对话质量。FED在这两个水平上都与人类判断达到中度到高度相关。

英文摘要:

It is important to define meaningful and interpretable automatic evaluation metrics for open-domain dialog research. Standard language generation metrics have been shown to be ineffective for dialog. This paper introduces the FED metric (fine-grained evaluation of dialog), an automatic evaluation metric which uses DialoGPT, without any fine-tuning or supervision. It also introduces the FED dataset which is constructed by annotating a set of human-system and human-human conversations with eighteen fine-grained dialog qualities. The FED metric (1) does not rely on a ground-truth response, (2) does not require training data and (3) measures fine-grained dialog qualities at both the turn and whole dialog levels. FED attains moderate to strong correlation with human judgement at both levels.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑