arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新，休息一下吧，周末愉快~~
arXiv 2604.08571cs.LGcs.AIcs.CL
收藏

鲁棒推理基准

Robust Reasoning Benchmark

University of Toronto（多伦多大学）；Vector Institute（向量研究所）

Pavel Golikov, Evgenii Opryshko, Gennady Pekhimenko, Mark C. Jeffrey

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究提出鲁棒推理基准（RRB），通过13种确定性文本扰动评估8种前沿模型，发现Claude在面对变换提示时表现出异常拒绝行为，而开放权重模型在结构噪声下出现多种失败模式，如认知冲刷、分词崩溃和推理崩溃，导致平均准确率下降高达54%。研究进一步发现由模型自身推理链引起的注意力稀释问题，并提出Intra-Query Attention Dilution概念，表明中间推理步骤会污染标准密集注意力机制，未来架构需整合显式上下文重置以实现可靠推理。

摘要：

尽管大型语言模型（LLMs）在标准数学基准上表现优异，但其问题解决能力依赖于上下文和文本格式。我们引入鲁棒推理基准（RRB），该基准由13种确定性文本扰动组成，应用于2024年和2025年的AIME。评估8种最先进的模型后，发现前沿模型总体上具有较强的鲁棒性，但Claude在面对变换提示时表现出异常拒绝行为。开放权重推理模型在结构噪声下表现出多种失败模式（认知冲刷、分词崩溃和推理崩溃），在扰动下平均准确率下降高达54%，某些扰动甚至导致100%的准确率下降。我们进一步研究其中一种失败模式：由模型自身推理链引起的注意力稀释。通过要求模型在单一上下文窗口内依次解决多个独立数学问题，我们识别出Intra-Query Attention Dilution。从7B到120B参数的开放权重模型在后续问题上的准确率逐渐下降，表明中间推理步骤会污染标准密集注意力机制。我们主张，为了实现可靠的推理，未来架构需要在模型自身推理链中整合显式上下文重置，从而引发关于推理任务最佳粒度的开放研究问题。

英文摘要：

While Large Language Models (LLMs) achieve high performance on standard mathematical benchmarks, their problem-solving abilities depend on the context and textual formatting. We introduce the Robust Reasoning Benchmark (RRB), a pipeline of 13 deterministic textual perturbations applied to AIME 2024 and AIME 2025. Evaluating 8 state-of-the-art models, we find that frontier models are largely resilient, with the notable exception of Claude, which categorically refuses many transformed prompts. Open-weights reasoning models exhibit a range of failure modes under structural noise (cognitive thrashing, tokenization breakdown, and reasoning collapse), with up to 54% average accuracy drops across perturbations and up to 100% on some. We further study one of these failure modes in isolation: attention dilution caused by the model's own chain-of-thought. By tasking models with solving multiple independent mathematical problems sequentially within a single context window, we identify Intra-Query Attention Dilution. Open-weights models ranging from 7B to 120B parameters exhibit accuracy decay on subsequent problems, suggesting that intermediate reasoning steps progressively pollute standard dense attention mechanisms. We argue that in order to achieve reliable reasoning, future architectures need to integrate explicit contextual resets within models' own chain-of-thought, leading to open research questions regarding the optimal granularity of reasoning tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑