推理扩展是否能提升推理的可靠性？对多模型自一致性权衡的分析
Does Inference Scaling Improve Reasoning Faithfulness? A Multi-Model Analysis of Self-Consistency Tradeoffs
AI总结：
研究探讨推理扩展对推理可靠性的影响，发现不同模型表现各异，自一致性并非普遍有效，需根据具体模型进行测试。
摘要：
自一致性已成为提高大语言模型在推理任务上的准确性的一种流行技术。该方法很简单：生成多个推理路径并通过多数投票选择最常见的答案。虽然这种方法能可靠地提高准确性，但尚不清楚这些提升是否反映了推理质量的真实改进。我们研究了一个此前未被探讨的基本问题：推理扩展是否能提升推理的可靠性？我们对四个前沿模型（GPT-5.2、Claude Opus 4.5、Gemini-3-flash-preview 和 DeepSeek-v3.2）在 100 个 GSM8K 数学推理问题上进行了全面的实证研究。我们的分析采用 bootstrap 置信区间、配对比较的 McNemar 测试和 Cohen's d 效应大小来严格量化效果。结果揭示了模型之间的显著差异，挑战了关于自一致性的一般假设。GPT-5.2 显示出预期的模式：准确性从 78% 提高到 90%（N=5），而可靠性相对稳定（0.540 到 0.510）。Claude Opus 4.5 则讲述了一个完全不同的故事。其准确性实际上从 78% 降至 74.3%，而可靠性从 0.270 跳跃到 0.891（N=5）。DeepSeek-v3.2 已经达到 98% 的准确性，显示出天花板效应，可靠性提升有限（0.440 到 0.541）。Gemini-3-flash 从 81% 提高到 86% 准确性，但可靠性略有下降（0.260 到 0.212）。问题难度分析显示，GPT-5.2 解决了 82% 的难题，而只破坏了 13% 的简单问题。相比之下，Claude 破坏了 23% 的简单问题，解释了其准确性下降。这些发现对从业者很重要：自一致性并非普遍有益，团队应在部署前测试其特定模型。我们发布了代码并提供了实际建议，以应对这些权衡。
英文摘要：
Self-consistency has emerged as a popular technique for improving large language model accuracy on reasoning tasks. The approach is straightforward: generate multiple reasoning paths and select the most common answer through majority voting. While this reliably boosts accuracy, it remains unclear whether these gains reflect genuine improvements in reasoning quality. We investigate a fundamental question that has not been studied before: does inference scaling improve reasoning faithfulness? We conduct a comprehensive empirical study across four frontier models (GPT-5.2, Claude Opus 4.5, Gemini-3-flash-preview, and DeepSeek-v3.2) on 100 GSM8K mathematical reasoning problems. Our analysis employs bootstrap confidence intervals, McNemar's tests for paired comparisons, and Cohen's d effect sizes to quantify the effects rigorously. The results reveal striking differences across models that challenge common assumptions about self-consistency. GPT-5.2 shows the expected pattern: accuracy improves from 78% to 90% at N=5, with faithfulness remaining relatively stable (0.540 to 0.510). Claude Opus 4.5 tells a completely different story. Its accuracy actually drops from 78% to 74.3% while faithfulness jumps dramatically from 0.270 to 0.891 at N=5. DeepSeek-v3.2, already at 98% accuracy, shows ceiling effects with modest faithfulness gains (0.440 to 0.541). Gemini-3-flash improves from 81% to 86% accuracy with a slight faithfulness decrease (0.260 to 0.212). Problem difficulty analysis reveals that GPT-5.2 solves 82% of hard problems while breaking only 13% of easy ones. Claude, in contrast, breaks 23% of easy problems, explaining its accuracy decrease. These findings matter for practitioners: self-consistency is not universally beneficial, and teams should test their specific models before deployment. We release our code and provide practical recommendations for navigating these tradeoffs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
