arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00564cs.AIcs.CL
收藏

揭示推理过程：一种评估大语言模型结构性数学推理的过程感知基准

Unmasking Reasoning Processes: A Process-aware Benchmark for Evaluating Structural Mathematical Reasoning in LLMs

Alibaba Group（阿里巴巴集团）；Shanghai Jiao Tong University（上海交通大学）
Xiang Zheng, Weiqi Zhai, Wei Wang, Boyu Yang, Wenbo Li, Ruixiang Luo, Haoxiang Sun, Yucheng Wang, Zhengze Li, Meng Wang, Yuetian Du, Guojie Lin, Yaxuan Wang, Xi… 展开作者

Xiang Zheng, Weiqi Zhai, Wei Wang, Boyu Yang, Wenbo Li, Ruixiang Luo, Haoxiang Sun, Yucheng Wang, Zhengze Li, Meng Wang, Yuetian Du, Guojie Lin, Yaxuan Wang, Xiaoxiao Xu, Yanhu Mo, Xuan Ren, Hu Wei, Bing Zhao

备注：
8 pages, and 3 figures
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出ReasoningMath-Plus基准，通过过程感知评估方法揭示大语言模型在结构性数学推理中的不足，引入HCRS评分函数和过程奖励模型，发现仅答案指标可能高估推理能力。

摘要：

近年来，大型语言模型（LLMs）在许多已建立的数学推理基准上实现了接近饱和的准确性，这引发了对其诊断真实推理能力的担忧。这种饱和性主要源于现有数据集中模板化计算和浅层算术分解的主导地位，这在一定程度上低估了多约束协调、构造性逻辑综合和空间推理等推理技能。为了解决这一差距，我们引入了ReasoningMath-Plus，一个包含150个精心挑选问题的基准，专门用于评估结构性推理。每个问题都强调在相互作用的约束下进行推理、构造性解决方案的形成或非平凡的结构洞察，并且附带了最小的推理骨架以支持细粒度的过程级评估。除了数据集外，我们还引入了HCRS（危险意识链式规则评分），一种确定性的步骤级评分函数，并在注释的推理轨迹上训练了一个过程奖励模型（PRM）。经验上，尽管领先模型在最终答案准确性上相对较高（最高达到5.8/10），但基于HCRS的总体评估得分却显著较低（平均4.36/10，最佳5.14/10），表明仅答案的指标可能高估了推理的稳健性。

英文摘要：

Recent large language models (LLMs) achieve near-saturation accuracy on many established mathematical reasoning benchmarks, raising concerns about their ability to diagnose genuine reasoning competence. This saturation largely stems from the dominance of template-based computation and shallow arithmetic decomposition in existing datasets, which underrepresent reasoning skills such as multi-constraint coordination, constructive logical synthesis, and spatial inference. To address this gap, we introduce ReasoningMath-Plus, a benchmark of 150 carefully curated problems explicitly designed to evaluate structural reasoning. Each problem emphasizes reasoning under interacting constraints, constructive solution formation, or non-trivial structural insight, and is annotated with a minimal reasoning skeleton to support fine-grained process-level evaluation. Alongside the dataset, we introduce HCRS (Hazard-aware Chain-based Rule Score), a deterministic step-level scoring function, and train a Process Reward Model (PRM) on the annotated reasoning traces. Empirically, while leading models attain relatively high final-answer accuracy (up to 5.8/10), HCRS-based holistic evaluation yields substantially lower scores (average 4.36/10, best 5.14/10), showing that answer-only metrics can overestimate reasoning robustness.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑