arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.16675cs.AI
收藏

LinAlg-Bench:一个 forensic 验证基准,揭示 LLM 数学推理中的结构失效模式

LinAlg-Bench: A Forensic Benchmark Revealing Structural Failure Modes in LLM Mathematical Reasoning

Department of Nuclear Engineering and Computer Science(核工程与计算机科学系)

Shradha Agarwal, Deepak Rajbhar, Tariq J

AI总结:

LinAlg-Bench 评估 10 个前沿大语言模型在结构线性代数计算中的表现,揭示 LLM 数学失败并非随机,而是受算法类型和矩阵维度约束。研究发现 4x4 尺寸存在行为阈值,低于该尺寸模型通过执行错误失败,高于则转向计算放弃,通过工具角色扮演等制造响应。

摘要:

我们介绍了 LinAlg-Bench,一个诊断基准,评估 10 个前沿大语言模型在结构线性代数计算中的表现,覆盖 3x3、4x4 和 5x5 矩阵的严格维度梯度。该基准涵盖 9 类任务和 660 个 SymPy 认证问题,评估 6,600 个模型输出。除了二元准确率外,LinAlg-Bench 引入了三阶段自动化取证流程,将 1,156 个失败分类为 10 个主要错误标签及其细粒度子类型,揭示 LLM 数学失败并非随机,而是受算法类型和矩阵维度约束。我们的核心发现是 4x4 尺寸存在行为阈值:低于该尺寸,模型通过执行错误失败——符号跟踪失败、算术漂移和奇偶错误;高于该尺寸,失败转变为计算放弃,模型通过工具角色扮演、约束一致的虚构和结构性幻觉制造响应而非尝试计算。这种制造到放弃的转变在所有模型层级和架构中几乎普遍存在,表明是工作记忆限制而非知识缺口,支持三种规模涌现的错误类型在 3x3 不存在但在 4x4 和 5x5 存在。我们进一步显示,解决方案策略的刚性是 5x5 确定性准确率的近完美预测因素,记录约束意识的虚构作为一种新的结构幻觉失败模式,并公开所有数据、模型输出、错误标签和判断流程。

英文摘要:

We introduce LinAlg-Bench, a diagnostic benchmark evaluating 10 frontier large language models on structured linear algebra computation across a strict dimensional gradient of 3x3, 4x4, and 5x5 matrices. Spanning 9 task types and 660 SymPy-certified problems, the benchmark exhaustively evaluates 6,600 model outputs. Beyond binary accuracy, LinAlg-Bench introduces a three-stage automated forensic pipeline classifying 1,156 failures into ten primary error tags with fine-grained subtypes, revealing that LLM mathematical failure is not random but structurally constrained by algorithm type and matrix dimension. Our central finding is a sharp behavioral threshold at 4x4 scale: below it, models fail through execution errors -- sign tracking failures, arithmetic drift, and parity errors; above it, failure transitions to computational abandonment, with models fabricating responses through tool roleplay, constraint-consistent confabulation, and structured hallucination rather than attempting computation. This fabrication-to-abandonment transition is near-universal across all model tiers and architectures, suggesting a working memory limit rather than a knowledge gap, supported by three scale-emergent error types absent at 3x3 but present at 4x4 and 5x5. We further show that solution strategy rigidity is a near-perfect predictor of 5x5 determinant accuracy, document constraint-aware confabulation as a novel structured hallucination failure mode, and release all data, model outputs, error labels, and judge pipeline publicly.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑