SLMJury：小型语言模型能否像大型模型一样进行评判？
SLMJury: Can Small Language Models Judge as Well as Large Ones?
AI总结：
提出SLMJury框架，评估小型语言模型作为评判者的能力，发现领域依赖的过度思考效应、领域泛化差异、闭端与开端评判能力分离，以及多智能体辩论降低准确性。
摘要：
大型语言模型（LLMs）被广泛用作评估模型输出的评判者，但其高成本、延迟和不透明性限制了可扩展性。我们引入SLMJury，一个评估小型语言模型（SLMs）作为评判者的框架，涵盖两种范式：闭端二元正确性和开端质量评分。我们在四个模型家族的16个SLM评判者（0.6B-14B参数）上，跨十个基准进行基准测试：八个闭端任务涵盖数学、科学和通用推理（每个配置N=64,824个判断），以及用于摘要和对话评分的SummEval和MT-Bench。我们将评判形式化为预算条件函数，并研究五个维度。得出四个发现。（1）过度思考效应是领域依赖的：对于大多数评判者，快速10令牌判决在数学评判上匹配或优于扩展推理（在有帮助的情况下提升2-7%），而推理在通用任务上胜出高达23%。（2）领域泛化区分了模型家族，数学到通用准确率差距从低于10%到接近40%不等。（3）闭端和开端评判依赖不同的能力：最佳二元评判者（Phi-4）在MT-Bench上降至第9名，而经过推理训练的模型则反转了这一顺序。（4）在反思-批判-改进（RCR）辩论协议下，多智能体辩论在所有测试配置中降低了准确性，而顶级评判者抵抗六种对抗性人格的方差<=0.55%。可靠的自动评估不需要大型专有模型，但没有单一的SLM占主导地位。排行榜可在https://anishh15.github.io/SLMJury/获取，我们的框架代码和pip包公开在https://github.com/anishh15/SLMJury和https://pypi.org/project/slmjury/。
英文摘要：
Large language models (LLMs) are widely used as judges for evaluating model outputs, but their high cost, latency, and opacity limit scalability. We introduce SLMJury, a framework for evaluating small language models (SLMs) as judges across two paradigms: closed-ended binary correctness and open-ended quality scoring. We benchmark 16 SLM judges (0.6B-14B parameters) from four model families across ten benchmarks: eight closed-ended tasks spanning mathematical, scientific, and general reasoning (N=64,824 judgments per configuration), plus SummEval and MT-Bench for summarization and conversational scoring. We formalize judging as a budget-conditioned function and study five dimensions. Four findings emerge. (1) The overthinking effect is domain-dependent: for most judges quick 10-token verdicts match or beat extended reasoning on mathematical judging (by 2-7% where they help), while reasoning wins on general tasks by up to 23%. (2) Domain generalization separates model families, with math-to-general accuracy gaps ranging from under 10% to nearly 40%. (3) Closed-ended and open-ended judging draw on different capabilities: the best binary judge (Phi-4) drops to rank 9 on MT-Bench, while reasoning-trained models invert this ordering. (4) Under the Reflect-Critique-Refine (RCR) debate protocol, multi-agent debate degrades accuracy across all tested configurations, whereas the top judges resist six adversarial personas with <=0.55% variance. Reliable automated evaluation does not require large proprietary models, yet no single SLM dominates. The leaderboard is available at https://anishh15.github.io/SLMJury/, and our framework code and pip package are publicly available at https://github.com/anishh15/SLMJury and https://pypi.org/project/slmjury/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
