推理链长度如何影响LLM对答案事实性的判断
How Long Reasoning Chains Influence LLMs' Judgment of Answer Factuality
AI总结：
研究探讨了推理链对LLM判断答案事实性的影响，发现弱判官易受推理存在影响，而强判官部分利用推理作为证据，但仍易被高质量推理链误导。
摘要：
大型语言模型（LLM）已被广泛用于人类评估的可扩展替代品，但此类判官仍不完美且易受表面偏见影响。可能原因是这些判官在评估答案正确性时缺乏足够信息。随着能推理的模型兴起，将生成器的推理内容暴露给判官能提供更丰富的信息，是提高判断准确性的自然候选。然而，其对判官行为的实际影响尚不明确。本文系统研究了访问推理链如何影响基于LLM的判断，涵盖事实性问答（QA）和数学推理基准。发现弱判官易受推理存在影响，频繁接受伴随流畅推理的错误答案，而强判官可部分利用推理作为信息性证据。然而，即使强判官也易被看似高质量的推理链误导。受控实验进一步揭示，推理链的流畅性和事实性是驱动判官决策的关键信号。这些发现突显了需要更鲁棒的LLM判官，以在评估现代推理模型时区分真实推理质量与表面流畅性。
英文摘要：
Large language models (LLMs) has been widely adopted as a scalable surrogate for human evaluation, yet such judges remain imperfect and susceptible to surface-level biases. One possible reason is that these judges lack sufficient information in assessing answer correctness. With the rise of reasoning-capable models, exposing a generator's reasoning content to the judge provides richer information and is a natural candidate for improving judgment accuracy. However, its actual impact on judge behavior remains understudied. In this paper, we systematically investigate how access to reasoning chains affects LLM-based judgment across factual question answering (QA) and mathematical reasoning benchmarks. We find that weak judges are easily swayed by reasoning presence, frequently accepting incorrect answers accompanied by fluent reasoning, while strong judges can partially leverage reasoning as informative evidence. Nevertheless, even strong judges are misled by seemingly high-quality reasoning chains. Controlled experiments further reveal that both fluency and factuality of reasoning chains are critical signals driving judge decisions. These findings highlight the need for more robust LLM judges that can distinguish genuine reasoning quality from superficial fluency when evaluating modern reasoning models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
