高效推理的艺术：数据、奖励与优化
The Art of Efficient Reasoning: Data, Reward, and Optimization
AI总结：
本文系统研究了大型语言模型的高效推理机制，通过精细化指标评估发现训练过程分为长度适应与推理优化两阶段，提出保持正奖励密度以避免短即正确陷阱，并验证了在不同领域和难度下的通用性。
摘要：
大型语言模型（LLMs）在扩展链式推理（CoT）方面表现良好，但也面临计算开销大的问题。为解决此问题，高效推理旨在激励短而准确的推理轨迹，通常通过强化学习（RL）中的奖励塑造实现。本文系统研究了LLM的高效推理机制。为全面评估，我们倡导更精细的指标，包括正确性条件下的长度分布和在2k至32k token预算范围内的性能表现。首先，我们发现训练过程遵循两阶段范式：长度适应与推理优化。通过大量实验（约0.2百万GPU小时）在统一协议下，我们分解了训练提示和回放、奖励塑造和优化策略。核心发现是保持足够的正奖励信号密度并避免短即正确陷阱。此外，学习到的长度偏差在不同领域和难度级别上具有通用性。我们将这些发现提炼为有价值的见解和实用指南，并在Qwen3模型（从0.6B到30B）上验证了其鲁棒性和通用性。权重可在https://wutaiqiang.github.io/project/Art获取。
英文摘要：
Large Language Models (LLMs) consistently benefit from scaled Chain-of-Thought (CoT) reasoning, but also suffer from heavy computational overhead. To address this issue, efficient reasoning aims to incentivize short yet accurate thinking trajectories, typically through reward shaping with Reinforcement Learning (RL). In this paper, we systematically investigate the mechanics of efficient reasoning for LLMs. For comprehensive evaluation, we advocate for more fine-grained metrics, including length distribution conditioned on correctness and performance across a wide spectrum of token budgets ranging from 2k to 32k. First, we reveal that the training process follows a two-stage paradigm: length adaptation and reasoning refinement. Through extensive experiments (about 0.2 million GPU hours) in a unified protocol, we deconstruct training prompts and rollouts, reward shaping, and optimization strategies. A central finding is to maintain a sufficient density of positive reward signals and avoid the short-is-correct trap. Moreover, the learned length bias generalizes across domains and difficulty levels. We distill these findings into valuable insights and practical guidelines, and validate them across the Qwen3 models ranging from 0.6B to 30B, demonstrating the robustness and generalization. Weights are available at https://wutaiqiang.github.io/project/Art
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
