显式推理使评判更可靠：对准确性、效率和鲁棒性的系统研究
Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness
AI总结：
本文通过对比显式推理与非显式推理模型在RewardBench任务中的表现，发现显式推理模型在准确性、效率和鲁棒性上均优于非显式模型，且在多语言环境下也表现出优势。
摘要：
随着大型语言模型（LLMs）在基准测试和奖励建模中被越来越多地用作自动评判者，确保其可靠性、效率和鲁棒性变得至关重要。本文通过使用开源的Qwen 3模型（参数规模为0.6B、1.7B和4B）系统地比较了“思考”和“非思考”LLM在LLM-as-a-judge范式中的表现。我们评估了RewardBench任务中的准确性和计算效率（FLOPs），并进一步检查了非思考模型的增强策略，包括上下文学习、规则引导评判、基于参考的评估和n-best聚合。结果显示，尽管这些增强措施，非思考模型在整体上仍无法达到思考模型的水平。我们的结果表明，思考模型在准确性上比非思考模型高出约10个百分点，且计算开销很小（不到2倍），相比之下，增强策略如少样本学习虽然在准确性上有小幅提升，但计算开销更大（超过8倍）。偏见和鲁棒性分析进一步表明，思考模型在各种偏见条件下（如位置偏见、从众偏见、身份偏见、多样性偏见和随机偏见）保持了显著更高的一致性（平均高出6%）。我们还扩展了实验到多语言环境，结果证实了显式推理的优势不仅限于英语。总体而言，本文的研究结果提供了系统证据，表明显式推理在LLM-as-a-judge范式中不仅在准确性和效率上，还在鲁棒性上具有明显优势。
英文摘要：
As Large Language Models (LLMs) are increasingly adopted as automated judges in benchmarking and reward modeling, ensuring their reliability, efficiency, and robustness has become critical. In this work, we present a systematic comparison of "thinking" and "non-thinking" LLMs in the LLM-as-a-judge paradigm using open-source Qwen 3 models of relatively small sizes (0.6B, 1.7B, and 4B parameters). We evaluate both accuracy and computational efficiency (FLOPs) on RewardBench tasks, and further examine augmentation strategies for non-thinking models, including in-context learning, rubric-guided judging, reference-based evaluation, and n-best aggregation. Our results show that despite these enhancements, non-thinking models generally fall short of their thinking counterparts. Our results show that thinking models achieve approximately 10% points higher accuracy with little overhead (under 2x), in contrast to augmentation strategies like few-shot learning, which deliver modest gains at a higher cost (>8x). Bias and robustness analyses further demonstrate that thinking models maintain significantly greater consistency under a variety of bias conditions such as positional, bandwagon, identity, diversity, and random biases (6% higher on average). We further extend our experiments to the multilingual setting and our results confirm that explicit reasoning extends its benefits beyond English. Overall, our work results in several important findings that provide systematic evidence that explicit reasoning offers clear advantages in the LLM-as-a-judge paradigm not only in accuracy and efficiency but also in robustness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
