arXiv 2303.040482023-10-24cs.CLcs.AI

ChatGPT 是一个好的 NLG 评估器吗？一项初步研究

Is ChatGPT a Good NLG Evaluator? A Preliminary Study

Soochow University（苏州大学）；Beijing Jiaotong University（北京交通大学）；Pattern Recognition Center, WeChat AI, Tencent Inc（腾讯公司微信AI模式识别中心）；Waseda University（早稻田大学）

Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, Jie Zhou

备注：: Both first authors contributed equally. Technical Report, 11 pages. Accepted to the 4th New Frontiers in Summarization Workshop (NewSumm@EMNLP 2023)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究对 ChatGPT 作为自然语言生成（NLG）评估指标的可靠性进行了初步元评估，发现其在五个数据集上多数情况下与人类判断的相关性达到最先进或具竞争力水平，但在依赖参考文本而存在偏差的数据集上效果会降低。

摘要：

最近，ChatGPT 的出现引起了计算语言学界的广泛关注。许多先前的研究表明，就自动评估指标而言，ChatGPT 在各种 NLP 任务上取得了显著的性能。然而，ChatGPT 作为评估指标的能力仍未得到充分探索。考虑到评估自然语言生成（NLG）模型的质量是一项艰巨的任务，且 NLG 指标众所周知地表现出与人类判断的较差相关性，我们想知道 ChatGPT 是否是一个好的 NLG 评估指标。在本报告中，我们对 ChatGPT 进行了初步的元评估，以展示其作为 NLG 指标的可靠性。具体而言，我们将 ChatGPT 视为人类评估器，并给出任务特定（例如，摘要）和方面特定（例如，相关性）的指令，以提示 ChatGPT 评估 NLG 模型的生成结果。我们在五个 NLG 元评估数据集（包括摘要、故事生成和数据到文本任务）上进行了实验。实验结果表明，与先前的自动指标相比，在大多数情况下，ChatGPT 实现了最先进的或具有竞争力的与人类判断的相关性。此外，我们发现 ChatGPT 评估器的有效性可能会受到元评估数据集创建方法的影响。对于严重依赖参考文本而创建从而存在偏差的元评估数据集，ChatGPT 评估器可能会失去其有效性。我们希望我们的初步研究能够推动通用可靠的 NLG 指标的出现。

英文摘要：

Recently, the emergence of ChatGPT has attracted wide attention from the computational linguistics community. Many prior studies have shown that ChatGPT achieves remarkable performance on various NLP tasks in terms of automatic evaluation metrics. However, the ability of ChatGPT to serve as an evaluation metric is still underexplored. Considering assessing the quality of natural language generation (NLG) models is an arduous task and NLG metrics notoriously show their poor correlation with human judgments, we wonder whether ChatGPT is a good NLG evaluation metric. In this report, we provide a preliminary meta-evaluation on ChatGPT to show its reliability as an NLG metric. In detail, we regard ChatGPT as a human evaluator and give task-specific (e.g., summarization) and aspect-specific (e.g., relevance) instruction to prompt ChatGPT to evaluate the generated results of NLG models. We conduct experiments on five NLG meta-evaluation datasets (including summarization, story generation and data-to-text tasks). Experimental results show that compared with previous automatic metrics, ChatGPT achieves state-of-the-art or competitive correlation with human judgments in most cases. In addition, we find that the effectiveness of the ChatGPT evaluator might be influenced by the creation method of the meta-evaluation datasets. For the meta-evaluation datasets which are created greatly depending on the reference and thus are biased, the ChatGPT evaluator might lose its effectiveness. We hope our preliminary study could prompt the emergence of a general-purposed reliable NLG metric.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。