arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.03550cs.AI
收藏

ReEfBench: 量化大语言模型的推理效率

ReEfBench: Quantifying the Reasoning Efficiency of LLMs

Westlake University(西湖大学);Imperial College London(帝国理工学院);New York University(纽约大学);Hainan University(海南大学)

Zhizhang Fu, Yuancheng Gu, Chenkai Hu, Hanmeng Liu, Yue Zhang

AI总结:

ReEfBench通过神经符号框架量化LLM推理效率,揭示推理性能提升源于真实推理而非冗长性,并指出训练中混合长短CoT数据的风险及蒸馏模型的局限性。

摘要:

测试时扩展性使大型语言模型(LLMs)能够处理复杂的推理任务,但当前的链式思维(CoT)评估方法的局限性模糊了性能提升是源于真正的推理还是仅仅是冗长性。为解决这一问题,(1)我们提出了一种新的神经符号框架,用于非侵入性、全面的过程导向评估。 (2)通过这一视角,我们识别出四个不同的行为原型并诊断失败模式。 (3)我们研究了推理模式、训练策略和模型规模的影响。我们的分析表明,扩展的标记生成不是深度推理的必要条件。此外,我们揭示了关键限制:在训练中混合长和短的CoT数据会带来提前饱和和崩溃的风险,而将模型蒸馏到更小的模型中会捕获行为长度,但因内在容量限制而无法复制逻辑有效性。

英文摘要:

Test-time scaling has enabled Large Language Models (LLMs) to tackle complex reasoning, yet the limitations of current Chain-of-Thought (CoT) evaluation obscures whether performance gains stem from genuine reasoning or mere verbosity. To address this, (1) we propose a novel neuro-symbolic framework for the non-intrusive, comprehensive process-centric evaluation of reasoning. (2) Through this lens, we identify four distinct behavioral prototypes and diagnose the failure modes. (3) We examine the impact of inference mode, training strategy, and model scale. Our analysis reveals that extended token generation is not a prerequisite for deep reasoning. Furthermore, we reveal critical constraints: mixing long and short CoT data in training risks in premature saturation and collapse, while distillation into smaller models captures behavioral length but fails to replicate logical efficacy due to intrinsic capacity limits.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑