arXiv 2601.225882026-07-10cs.CLcs.AIcs.LG

重新思考大语言模型作为评判器：通过语义能力不对称利用小语言模型进行表示作为评判器

Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry

Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）；University of Pittsburgh（匹兹堡大学）；University of Maryland, College Park（马里兰大学学院公园分校）；University of Connecticut（康涅狄格大学）

Zhuochun Li, Yong Zhang, Ming Li, Yuelyu Ji, Yiming Zeng, Ning Cheng, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao, Daqing He

发表信息：: ICLR 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究探讨小语言模型能否作高效评估器，基于语义能力不对称假设提出“表示作为评判器”范式，通过INSPECTOR框架实例化，实验显示其在推理基准测试中性能出色，为可扩展评估提供了更优选择。

摘要：

大语言模型（LLMs）被广泛用作无需参考的评估器，但“LLM作为评判器”范式成本高、不透明且对提示设计敏感。本文研究较小模型能否通过利用内部表示而非表面生成来作为高效评估器。发现小语言模型虽生成能力弱，但隐藏状态中编码了丰富评估信号，由此提出语义能力不对称假设。基于此从“LLM作为评判器”转变为“表示作为评判器”，通过INSPECTOR框架实例化该范式，实验表明其性能优于基于提示的小语言模型且接近完整LLM评判器，提供了更高效、可靠和可解释的评估方式。

英文摘要：

Large language models (LLMs) are widely used as reference-free evaluators via prompting, but this "LLM-as-a-Judge" paradigm is costly, opaque, and sensitive to prompt design. In this work, we investigate whether smaller models can serve as efficient evaluators by leveraging internal representations instead of surface generation. We uncover a consistent empirical pattern: small LMs, despite with weak generative ability, encode rich evaluative signals in their hidden states. This motivates us to propose the Semantic Capacity Asymmetry Hypothesis: evaluation requires significantly less semantic capacity than generation and can be grounded in intermediate representations, suggesting that evaluation does not necessarily need to rely on large-scale generative models but can instead leverage latent features from smaller ones. Our findings motivate a paradigm shift from LLM-as-a-Judge to Representation-as-a-Judge, a decoding-free evaluation strategy that probes internal model structure rather than relying on prompted output. We instantiate this paradigm through INSPECTOR, a probing-based framework that predicts aspect-level evaluation scores from small model representations. Experiments on reasoning benchmarks (GSM8K, MATH, GPQA) show that INSPECTOR substantially outperforms prompting-based small LMs and closely approximates full LLM judges, while offering a more efficient, reliable, and interpretable alternative for scalable evaluation. The code and data are available at: https://github.com/zhuochunli/Representation-as-a-judge

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。