通过渐进性思维编码高效训练大型推理模型
Training Large Reasoning Models Efficiently via Progressive Thought Encoding
AI总结：
通过渐进性思维编码方法，提升大型推理模型在有限内存下的推理效率和准确性。
摘要：
大型推理模型（LRMs）在复杂问题上表现出色，但面临效率的关键障碍：强化学习（RL）训练需要长滚出以获得基于结果的奖励，其中自回归解码主导了时间和内存的使用。虽然滑动窗口缓存策略可以限制内存，但它们会破坏长上下文推理并降低性能。我们引入了渐进性思维编码，这是一种参数高效微调方法，使LRMs能够在固定大小的缓存中有效推理。通过逐步将中间推理编码为固定大小的向量表示，我们的方法消除了需要通过完整缓存滚出进行反向传播的必要，从而减少内存使用，同时在推理期间保持恒定的内存。在三个模型上进行了实验，包括Qwen2.5-3B-Instruct、Qwen2.5-7B-Instruct和DeepSeek-R1-Distill-Llama-8B，在六个广泛使用的具有挑战性的数学基准测试中，结果显示出一致的提升：我们的方法在LoRA基于微调的基础上实现了+19.3%的改进，并在平均情况下比未微调的LRMs实现了+29.9%的改进，在相同的紧绷缓存预算下，最高可提高AIME2024/2025的准确率达23.4%。这些结果表明，渐进性思维编码不仅提高了推理准确性，还使LRMs的RL训练在现实世界的内存限制下变得显著更高效和可扩展。
英文摘要：
Large reasoning models (LRMs) excel on complex problems but face a critical barrier to efficiency: reinforcement learning (RL) training requires long rollouts for outcome-based rewards, where autoregressive decoding dominates time and memory usage. While sliding-window cache strategies can bound memory, they disrupt long-context reasoning and degrade performance. We introduce Progressive Thought Encoding, a parameter-efficient fine-tuning method that enables LRMs to reason effectively under fixed-size caches. By progressively encoding intermediate reasoning into fixed-size vector representations, our approach eliminates the need to backpropagate through full-cache rollouts, thereby reducing memory usage, while maintaining constant memory during inference. Experiments on three models, including Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and DeepSeek-R1-Distill-Llama-8B, on six widely used challenging mathematical benchmarks show consistent gains: our method achieves +19.3% improvement over LoRA-based fine-tuning and +29.9% over LRMs without fine-tuning on average, with up to +23.4 accuracy improvement on AIME2024/2025 under the same tight cache budgets. These results demonstrate that Progressive Thought Encoding not only improves reasoning accuracy but also makes RL training of LRMs substantially more efficient and scalable under real-world memory constraints.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
