大型语言模型作为自动对话评估器有效性的综合分析
A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators
AI总结：
本文系统评估30个LLM在12个元评估数据集上的话轮级与对话级多维评分、对抗鲁棒性及集成效果，以检验其作为自动对话评估器的有效性。
摘要：
自动评估是对话系统研究中不可或缺的一环。传统的基于参考文本的NLG指标通常被认为不适合对话评估。因此，近期研究提出了多种独特的、无参考的神经指标，它们与人类评估更为一致。其中值得注意的是，大型语言模型（LLM），尤其是像ChatGPT这样的指令微调变体，被证明是人类评判者的有希望的替代者。然而，现有利用LLM进行自动对话评估的工作在元评估数据集数量、评估模式、LLM覆盖范围等方面范围有限。因此，这些LLM究竟有多有效仍无定论。为此，我们对LLM在自动对话评估中的应用开展了综合研究。具体而言，我们使用包含12个元评估数据集的全面集合，分析了近期出现的30个LLM在话轮和对话两个层面的多维评估能力。此外，我们探究了LLM在话轮和对话层面处理各类对抗性扰动时的鲁棒性。最后，我们探索了模型级和维度级集成如何影响评估性能。所有资源均可在https://github.com/e0397123/comp-analysis获取。
英文摘要：
Automatic evaluation is an integral aspect of dialogue system research. The traditional reference-based NLG metrics are generally found to be unsuitable for dialogue assessment. Consequently, recent studies have suggested various unique, reference-free neural metrics that better align with human evaluations. Notably among them, large language models (LLMs), particularly the instruction-tuned variants like ChatGPT, are shown to be promising substitutes for human judges. Yet, existing works on utilizing LLMs for automatic dialogue evaluation are limited in their scope in terms of the number of meta-evaluation datasets, mode of evaluation, coverage of LLMs, etc. Hence, it remains inconclusive how effective these LLMs are. To this end, we conduct a comprehensive study on the application of LLMs for automatic dialogue evaluation. Specifically, we analyze the multi-dimensional evaluation capability of 30 recently emerged LLMs at both turn and dialogue levels, using a comprehensive set of 12 meta-evaluation datasets. Additionally, we probe the robustness of the LLMs in handling various adversarial perturbations at both turn and dialogue levels. Finally, we explore how model-level and dimension-level ensembles impact the evaluation performance. All resources are available at https://github.com/e0397123/comp-analysis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
