arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.00552cs.CL
收藏

捉摸不到:为何小型推理模型用数学严谨性伪装推理

Catch Me If You Can: How Smaller Reasoning Models Pretend to Reason with Mathematical Fidelity

Subramanyam Sahoo, Vinija Jain, Saanidhya Vats, Siddharth Mohapatra, Rui Min, Aman Chadha, Divya Chaudhary

AI总结:

该研究提出诊断框架,揭示小型模型依赖模式匹配而非真实逻辑计算,通过四个轴线评估推理忠实性,推动可验证的数学推理研究。

摘要:

当前对语言模型数学推理能力的评估主要依赖于答案准确性,这可能掩盖了逻辑计算中的根本性失败。我们引入了一个诊断框架,通过四个互补的轴线来区分真正的数学推理与表面的模式匹配:前向-后向一致性、传递性覆盖、反事实敏感性和扰动鲁棒性。通过将此框架应用于MenatQA数据集上的Qwen3-0.6B模型进行案例研究,我们揭示了表面性能与推理忠实性之间的显著脱节。虽然该模型在答案准确性上表现合理(70%以上),但其后向一致性较差(15%),传递性覆盖有限(32.2%),并且对扰动敏感性脆弱。我们的诊断揭示了传统准确性指标无法检测到的推理失败,表明该小型模型严重依赖于模式匹配而非真正的逻辑计算。尽管我们的实证发现基于一个600M参数模型,但该诊断框架本身是模型无关的,并具有通用性。我们发布了评估协议,以使研究社区能够评估不同模型规模和架构下的推理忠实性,从而超越表面准确性,迈向可验证的数学推理。

英文摘要:

Current evaluation of mathematical reasoning in language models relies primarily on answer accuracy, potentially masking fundamental failures in logical computation. We introduce a diagnostic framework that distinguishes genuine mathematical reasoning from superficial pattern matching through four complementary axes: forward-backward consistency, transitivity coverage, counterfactual sensitivity, and perturbation robustness. Through a case study applying this framework to Qwen3-0.6B on the MenatQA dataset, we reveal a striking disconnect between surface performance and reasoning fidelity. While the model achieves reasonable answer accuracy (70%+), it demonstrates poor backward consistency (15%), limited transitivity coverage (32.2%), and brittle sensitivity to perturbations. Our diagnostics expose reasoning failures invisible to traditional accuracy metrics, suggesting that this small model relies heavily on pattern matching rather than genuine logical computation. While our empirical findings are based on a single 600M-parameter model, the diagnostic framework itself is model-agnostic and generalizable. We release our evaluation protocols to enable the research community to assess reasoning fidelity across different model scales and architectures, moving beyond surface-level accuracy toward verifiable mathematical reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑