既无效也不可靠？探究大语言模型作为评判者的使用
Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
AI总结：
本文从测量理论出发，批判性审视大语言模型作为评判者的四个核心假设，并结合文本摘要、数据标注和安全对齐三个应用，呼吁建立更负责任的评估实践。
摘要：
评估自然语言生成（NLG）系统仍然是自然语言处理（NLP）的核心挑战，而旨在通用的大语言模型（LLMs）的兴起进一步加剧了这一挑战。近年来，大语言模型作为评判者（LLJs）已成为传统指标的一种有前景的替代方案，但其有效性仍未得到充分探索。本立场论文认为，当前围绕 LLJs 的热情可能为时过早，因为其采用速度已超过对其作为评估者的可靠性和有效性的严格审视。借鉴社会科学中的测量理论，我们识别并批判性地评估了支撑 LLJs 使用的四个核心假设：其充当人类判断代理的能力、其作为评估者的能力、其可扩展性以及其成本效益。我们考察了这些假设中的每一个如何可能受到 LLMs、LLJs 的固有限制或当前 NLG 评估实践的挑战。为使分析落地，我们探讨了 LLJs 的三个应用：文本摘要、数据标注和安全对齐。最后，我们强调在 LLJs 评估中需要更负责任的评估实践，以确保其在领域中日益增长的作用能够支持而非损害 NLG 的进步。
英文摘要：
Evaluating natural language generation (NLG) systems remains a core challenge of natural language processing (NLP), further complicated by the rise of large language models (LLMs) that aims to be general-purpose. Recently, large language models as judges (LLJs) have emerged as a promising alternative to traditional metrics, but their validity remains underexplored. This position paper argues that the current enthusiasm around LLJs may be premature, as their adoption has outpaced rigorous scrutiny of their reliability and validity as evaluators. Drawing on measurement theory from the social sciences, we identify and critically assess four core assumptions underlying the use of LLJs: their ability to act as proxies for human judgment, their capabilities as evaluators, their scalability, and their cost-effectiveness. We examine how each of these assumptions may be challenged by the inherent limitations of LLMs, LLJs, or current practices in NLG evaluation. To ground our analysis, we explore three applications of LLJs: text summarization, data annotation, and safety alignment. Finally, we highlight the need for more responsible evaluation practices in LLJs evaluation, to ensure that their growing role in the field supports, rather than undermines, progress in NLG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
