arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2110.01895cs.CL
收藏

预训练语言模型对对话评估的影响研究

Investigating the Impact of Pre-trained Language Models on Dialog Evaluation

National University of Singapore(新加坡国立大学);Robert Bosch (SEA)(罗伯特博世(东南亚));Universidad Politécnica de Madrid(马德里理工大学);Kriston AI Lab(Kriston人工智能实验室)

Chen Zhang, Luis Fernando D'Haro, Yiming Chen, Thomas Friedrichs, Haizhou Li

摘要:

最近,有一个在自动开放域对话评估中应用预训练语言模型(Pr-LM)的兴趣激增。Pr-LM为解决多领域评估挑战提供了一个有前途的方向。然而,不同的Pr-LM对自动度量性能的影响还没有得到很好的理解。本文研究了8种不同的Pr-LM,并研究了它们对三种典型的自动对话评估指标的影响。具体来说,我们分析了如何选择的Pr-LM影响自动度量的性能。对每个指标进行广泛的相关性分析,以评估不同Pr-LM沿各个轴的影响,包括预训练目标,对话评估标准,模型大小和跨数据集鲁棒性。这项研究作为第一次全面评估不同的Pr-LM对自动对话评价的影响。

英文摘要:

Recently, there is a surge of interest in applying pre-trained language models (Pr-LM) in automatic open-domain dialog evaluation. Pr-LMs offer a promising direction for addressing the multi-domain evaluation challenge. Yet, the impact of different Pr-LMs on the performance of automatic metrics is not well-understood. This paper examines 8 different Pr-LMs and studies their impact on three typical automatic dialog evaluation metrics across three different dialog evaluation benchmarks. Specifically, we analyze how the choice of Pr-LMs affects the performance of automatic metrics. Extensive correlation analyses on each of the metrics are performed to assess the effects of different Pr-LMs along various axes, including pre-training objectives, dialog evaluation criteria, model size, and cross-dataset robustness. This study serves as the first comprehensive assessment of the effects of different Pr-LMs on automatic dialog evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑