arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新，休息一下吧，周末愉快~~
arXiv 2602.08672cs.CLcs.LG
收藏

学习评判：LLMs设计与应用评价量规

Learning to Judge: LLMs Designing and Applying Evaluation Rubrics

Centrum Wiskunde & Informatica (CWI)（荷兰阿姆斯特丹数学与信息学中心）

Clemencia Siro, Pourya Aliannejadi, Mohammad Aliannejadi

备注：
Accepted at EACL 2026 Findings
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文研究LLMs能否自主设计和应用评价量规，发现其在一致性与跨模型泛化上存在差异，需联合建模人类与LLM的评估语言以提升可靠性。

摘要：

大型语言模型（LLMs）越来越多地被用作自然语言生成的评估者，通过人类定义的评分标准来评估系统输出。然而，人类评分标准往往静态且与模型内部如何表示语言质量不一致。我们引入GER-Eval（生成评价评分标准以评估）来探讨LLMs是否能够设计和应用自己的评分标准。我们评估了LLMs定义的评分标准的语义连贯性和评分可靠性，以及其与人类评分标准的一致性。LLMs能够可靠地生成可解释且任务感知的评估维度，并在模型内部一致地应用这些维度，但其在事实性和知识密集型设置中的评分可靠性会下降。闭源模型如GPT-4o在一致性和跨模型泛化方面优于开源模型如Llama。我们的发现将评估定位为LLMs学习的语言能力，模型内部一致但跨模型碎片化，并呼吁新的方法，联合建模人类和LLM的评估语言以提高可靠性和可解释性。

英文摘要：

Large language models (LLMs) are increasingly used as evaluators for natural language generation, applying human-defined rubrics to assess system outputs. However, human rubrics are often static and misaligned with how models internally represent language quality. We introduce GER-Eval (Generating Evaluation Rubrics for Evaluation) to investigate whether LLMs can design and apply their own evaluation rubrics. We evaluate the semantic coherence and scoring reliability of LLM-defined criteria and their alignment with human criteria. LLMs reliably generate interpretable and task-aware evaluation dimensions and apply them consistently within models, but their scoring reliability degrades in factual and knowledge-intensive settings. Closed-source models such as GPT-4o achieve higher agreement and cross-model generalization than open-weight models such as Llama. Our findings position evaluation as a learned linguistic capability of LLMs, consistent within models but fragmented across them, and call for new methods that jointly model human and LLM evaluative language to improve reliability and interpretability.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑