我们离最优推理效率还有多远？
How Far Are We from Optimal Reasoning Efficiency?
AI总结：
本文提出推理效率前沿与推理效率差距（REG）指标，并设计REO-RL强化学习算法，在数学基准上显著降低REG，同时保持准确性，逼近最优推理效率。
摘要：
大型推理模型（LRMs）通过扩展的思维链（CoT）推理展现出卓越的问题解决能力，但往往会产生过度冗长和冗余的推理轨迹。这种低效性导致高昂的推理成本，并限制了实际部署。虽然现有的微调方法旨在提高推理效率，但由于评估不一致，评估其效率提升仍然具有挑战性。在这项工作中，我们引入了推理效率前沿，即通过对基础LRMs在不同方法和训练配置下进行微调而获得的经验上界。基于这些前沿，我们提出了推理效率差距（REG），一个量化任何微调后的LRMs偏离这些前沿程度的统一指标。在具有挑战性的数学基准上的系统评估揭示了当前方法的显著差距：它们要么牺牲准确性以换取较短长度，要么在严格的令牌预算下仍然效率低下。为了缩小效率差距，我们提出了REO-RL，一类通过针对稀疏的令牌预算集合最小化REG的强化学习算法。利用对策略性选择的预算进行数值积分，REO-RL使用少量令牌预算以低误差近似完整的效率目标。通过系统基准测试，我们证明了我们的效率指标REG有效捕捉了准确性与长度之间的权衡，低REG方法在保持准确性的同时减少了长度。我们的方法REO-RL在所有评估的LRMs上持续将REG降低至少50，并在16K令牌预算下以最小的准确性损失匹配Qwen3-4B/8B的效率前沿。消融研究证实了我们指数令牌预算策略的有效性。最后，我们的发现强调，微调LRMs以完美对齐效率前沿仍然是一个开放的挑战。
英文摘要：
Large Reasoning Models (LRMs) demonstrate remarkable problem-solving capabilities through extended Chain-of-Thought (CoT) reasoning but often produce excessively verbose and redundant reasoning traces. This inefficiency incurs high inference costs and limits practical deployment. While existing fine-tuning methods aim to improve reasoning efficiency, assessing their efficiency gains remains challenging due to inconsistent evaluations. In this work, we introduce the reasoning efficiency frontiers, empirical upper bounds derived from fine-tuning base LRMs across diverse approaches and training configurations. Based on these frontiers, we propose the Reasoning Efficiency Gap (REG), a unified metric quantifying deviations of any fine-tuned LRMs from these frontiers. Systematic evaluation on challenging mathematical benchmarks reveals significant gaps in current methods: they either sacrifice accuracy for short length or still remain inefficient under tight token budgets. To reduce the efficiency gap, we propose REO-RL, a class of Reinforcement Learning algorithms that minimizes REG by targeting a sparse set of token budgets. Leveraging numerical integration over strategically selected budgets, REO-RL approximates the full efficiency objective with low error using a small set of token budgets. Through systematic benchmarking, we demonstrate that our efficiency metric, REG, effectively captures the accuracy-length trade-off, with low-REG methods reducing length while maintaining accuracy. Our approach, REO-RL, consistently reduces REG by >=50 across all evaluated LRMs and matching Qwen3-4B/8B efficiency frontiers under a 16K token budget with minimal accuracy loss. Ablation studies confirm the effectiveness of our exponential token budget strategy. Finally, our findings highlight that fine-tuning LRMs to perfectly align with the efficiency frontiers remains an open challenge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
