arXiv 2601.036302026-05-14cs.CL

推理模型优于LLM-判官，但存在偏见

Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases

Institute of Science Tokyo（东京科学研究院）；Harbin Institute of Technology（哈尔滨工业大学）；The University of Osaka（大阪大学）

Hui Huang, Xuanxin Wu, Muyun Yang, Yuki Arase

备注：: Accepted by ACL 2026 Workshop EvalEval

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文系统比较了推理模型在判断任务中的表现，发现其在准确性、指令遵循和抗攻击能力上优于非推理LLM，但存在显著偏见，提出PlanJudge策略以缓解偏见问题。

摘要：

本文首次系统比较了大型推理模型（LRMs）是否优于非推理LLM在判断任务中的表现。实证分析得出四个关键发现：1）LRMs在判断准确性上优于非推理LLM，特别是在推理密集型任务中；2）LRMs在评估指令遵循能力上表现更优；3）LRMs对针对判断任务的对抗攻击更具鲁棒性；4）然而，LRMs仍然表现出强烈的评估偏见。为缓解此偏见漏洞，我们提出PlanJudge，一种轻量级评估策略，促使模型在执行判断前生成明确的评估计划。尽管其简单，实验表明PlanJudge显著减轻了LLM作为判官的偏见问题，同时保持了总体判断准确性。

英文摘要：

This paper presents the first systematic comparison investigating whether Large Reasoning Models (LRMs) are superior judges to non-reasoning LLMs. Our empirical analysis yields four key findings: 1) LRMs outperform non-reasoning LLMs in terms of judgment accuracy, particularly on reasoning-intensive tasks; 2) LRMs demonstrate superior evaluation instruction-following capabilities; 3) LRMs exhibit enhanced robustness against adversarial attacks targeting judgment tasks; 4) However, LRMs still exhibit strong evaluation biases. To mitigate this bias vulnerability, we propose PlanJudge, a lightweight evaluation strategy that prompts the model to generate an explicit evaluation plan before executing the judgment. Despite its simplicity, our experiments demonstrate that PlanJudge significantly mitigates biases in LLM-as-a-Judge while preserving overall judgment accuracy.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。