大语言模型是否会过度思考基础数学推理？评估语言模型中准确性与效率的权衡
Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
AI总结：
本文提出LLMThinkBench基准测试，评估语言模型在准确性与过度思考之间的权衡，发现模型在复杂基准上的表现不直接转化为基础数学推理能力，且过度思考导致效率下降。
摘要：
大型语言模型（LLMs）在复杂数学基准上表现出色，但在基础数学推理上有时会失败，并生成不必要的冗长响应。本文提出了LLMThinkBench，一个系统性的基准测试和全面的实证研究，旨在评估LLMs的推理效率，重点在于准确性与过度思考之间的基本权衡。首先，我们正式化了准确性与冗长性之间的权衡。其次，我们引入了过度思考得分，一种结合准确性和令牌效率的调和平均度量，用于全面评估模型。第三，我们建立了一个包含14个基础数学任务的评估协议，动态生成数据。第四，我们进行了大规模实证研究，评估了53个LLMs，包括推理和量化变体，不同推理预算下。第五，我们发布了LLMThinkBench作为开源Python包和公开排行榜以确保可重复性。我们的发现揭示：1）模型在复杂基准上的表现不直接转化为基础数学推理；2）推理模型生成约18倍更多的令牌，但有时准确率更低，当令牌受限时，会出现灾难性崩溃，准确率下降高达约36%；3）准确性与冗长性之间的关系是非单调的，随着推理预算的增加，收益递减（GPT-5/o系列模型在低到中等再到高推理努力中没有准确性提升）。我们的发现挑战了LLMs中更长推理必然提高数学推理的假设。公开排行榜可在https://ctrl-gaurav.github.io/LLMThinkBench/获取。开源Python包可在https://pypi.org/project/llmthinkbench/获取，代码库可在https://github.com/ctrl-gaurav/LLMThinkBench找到以方便和可重复评估。
英文摘要：
Large language models (LLMs) achieve impressive performance on complex mathematical benchmarks yet sometimes fail on basic math reasoning while generating unnecessarily verbose responses. In this paper, we present LLMThinkBench, a systematic benchmark and comprehensive empirical study to evaluate the efficiency of reasoning in LLMs, focusing on the fundamental tradeoff between accuracy and overthinking. First, we formalize the accuracy-verbosity tradeoff. Second, we introduce the Overthinking Score, a harmonic-mean metric combining accuracy and token-efficiency for holistic model evaluation. Third, we establish an evaluation protocol with dynamically-generated data across 14 basic math tasks. Fourth, we conduct a large-scale empirical study evaluating 53 LLMs, including reasoning and quantized variants across different reasoning budgets. Fifth, we release LLMThinkBench as an open-source Python package and public leaderboard for reproducibility. Our findings reveal: 1) model performance on complex benchmarks does not translate directly to basic math reasoning; 2) reasoning models generate ~18x more tokens while sometimes achieving lower accuracy and exhibit catastrophic collapse when tokens are constrained, dropping by up to ~36%; 3) the accuracy-verbosity relationship is non-monotonic with extended reasoning budgets yielding diminishing returns (GPT-5/o-series models show zero accuracy gain from low -> medium -> high reasoning effort). Our findings challenge the assumption that longer reasoning in LLMs necessarily improves mathematical reasoning. Our public leaderboard is available at https://ctrl-gaurav.github.io/LLMThinkBench/. Our open-source Python package is available at https://pypi.org/project/llmthinkbench/, and the codebase can be found at https://github.com/ctrl-gaurav/LLMThinkBench for easy and reproducible evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
