arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.07795cs.AIcs.CLcs.LG
收藏

ReasonBENCH: 基准测试LLM推理的(不)稳定性

ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning

Aarhus University(奥胡斯大学);Indian Institute of Technology Delhi(德里印度理工学院);EPFL(苏黎世联邦理工学院)

Nearchos Potamitis, Vansh Ramani, Har Ashish Arora, Dhairya Kuchhal, Lars Klein, Akhil Arora

AI总结:

提出ReasonBench基准套件,通过30次独立试验揭示LLM推理系统在贪婪解码下仍存在结构化方差,并引入全局噪声和运行噪声分类法,证明稳定性是推理系统的固有属性,倡导分布感知评估。

摘要:

LLM推理系统的基准分数被报告为单一数字,然而相同的模型、策略和任务在重复执行时,即使在贪婪解码(T=0)下也会产生显著不同的答案和成本。这种方差并非统计上的麻烦:性能最高的策略在与最接近的对手进行头对头运行时仅获胜77%,这意味着单次观测到的分数可能会无声地错误排序系统。我们引入了ReasonBench,一个基准套件,记录了10种推理策略、12个模型和6个任务的30次独立试验,将质量和成本视为分布而非点估计。我们发现这种方差是有结构的而非随机的:一个双组分分类法——全局噪声(捕捉跨基准的不均匀性)和运行噪声(捕捉基准内的随机性)——揭示了策略架构预测稳定性分布,而模型和策略则移动分布的正交方面。层次分解将四分之三的分数方差归因于基准、系统和项目结构,而单次运行评估无声地吸收了持久的残差。最后,成本和成本非对称地解耦:廉价方法在结构上对联合成本-质量失败免疫,而昂贵方法无论其准确性如何仍然暴露。这些发现确立了不稳定性作为推理系统的固有属性,并促使分布感知评估成为标准实践。

英文摘要:

Benchmark scores for LLM reasoning systems are reported as single numbers, yet the same model, strategy, and task can produce meaningfully different answers and costs across repeated executions, even under greedy decoding (T = 0). This variance is not a statistical nuisance: the highest-performing strategy wins only 77% of head-to-head runs against its nearest competitor, meaning a single observed score can silently misrank systems. We introduce ReasonBench, a benchmark suite recording 30 independent trials across 10 reasoning strategies, 12 models, and 6 tasks, treating quality and cost as distributions rather than point estimates. We find that this variance is structured rather than random: a two-component taxonomy -- Global Noise, capturing cross-benchmark unevenness, and Run Noise, capturing within-benchmark stochasticity -- reveals that strategy architecture predicts stability profiles, while models and strategies shift orthogonal aspects of the distribution. A hierarchical decomposition attributes three-quarters of score variance to benchmark, system, and item structure, with a persistent residual that single-run evaluation silently absorbs. Finally, cost and quality decouple asymmetrically: cheap methods are structurally immune to joint cost-quality failure, while expensive methods remain exposed regardless of their accuracy. These findings establish instability as an inherent property of reasoning systems and motivate distribution-aware evaluation as standard practice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑