arXiv 2406.184032025-06-02cs.CL

用 LLMs 代替人类评委？跨越 20 个 NLP 评估任务的大规模实证研究

LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks

University of Amsterdam（阿姆斯特丹大学）；University of Trento（特伦托大学）；University of Copenhagen（哥本哈根大学）；Utrecht University（乌得勒支大学）；Max Planck Institute for Psycholinguistics（马克斯·普朗克心理语言学研究所）；ETH Zürich（苏黎世联邦理工学院）；Saarland University（萨尔大学）；Universidade de Lisboa & Unbabel（里斯本大学及Unbabel）；LMU Munich & MCML（慕尼黑大学及MCML）；University of Potsdam（波茨坦大学）；Heriot-Watt University（赫瑞线学院）

Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexande… 展开作者

Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni

备注：: Accepted to the main conference of ACL 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出包含20个NLP数据集的JUDGE-BENCH，全面评估11个LLMs替代人类评委的标注复现能力，发现其可靠性因任务、属性和语言来源差异显著，建议使用前需严格验证。

摘要：

使用 LLMs 代替人类判断来评估 NLP 模型的趋势日益增长，这引发了关于这些评估有效性的问题，以及在专有模型情况下的可复现性问题。我们提供了 JUDGE-BENCH，这是一个包含 20 个 NLP 数据集的可扩展集合，带有覆盖广泛评估属性和数据类型的人类标注，并全面评估了 11 个当前的 LLMs（涵盖 open-weight 和专有模型）复制这些标注的能力。我们的评估显示，模型和数据集之间存在显著差异。模型在某些任务上是可靠的评估者，但总体而言，根据所评估的属性、人类评委的专业水平以及语言是人类生成还是模型生成的，表现出显著的变异性。我们得出结论，在将 LLMs 用作评估者之前，应仔细对照人类判断进行验证。

英文摘要：

There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments, raising questions about the validity of these evaluations, as well as their reproducibility in the case of proprietary models. We provide JUDGE-BENCH, an extensible collection of 20 NLP datasets with human annotations covering a broad range of evaluated properties and types of data, and comprehensively evaluate 11 current LLMs, covering both open-weight and proprietary models, for their ability to replicate the annotations. Our evaluations show substantial variance across models and datasets. Models are reliable evaluators on some tasks, but overall display substantial variability depending on the property being evaluated, the expertise level of the human judges, and whether the language is human or model-generated. We conclude that LLMs should be carefully validated against human judgments before being used as evaluators.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。