揭秘智能体推理中的强化学习
Demystifying Reinforcement Learning in Agentic Reasoning
AI总结：
本文从数据、算法和推理模式三方面揭秘智能体推理中的强化学习，提出真实端到端轨迹初始化、探索友好型技术与审慎策略，使4B模型超越32B模型。
摘要：
最近，智能体RL的出现表明RL也能有效提升LLMs的智能体推理能力，但关键设计原则和最佳实践仍不清晰。在这项工作中，我们从数据、算法和推理模式三个关键视角进行了全面系统的调查，以揭秘智能体推理中的强化学习。我们强调以下关键见解：(i)用真实的端到端工具使用轨迹替换拼接的合成轨迹，能产生强得多的SFT初始化；高多样性、模型感知的数据集能维持探索并显著提高RL性能。(ii)对智能体RL而言，探索友好型技术至关重要，例如clip higher、超长奖励塑造以及保持足够的策略熵可以提高训练效率。(iii)具有较少工具调用的审慎策略优于频繁的工具调用或冗长的自我推理，能提高工具效率和最终准确率。这些简单实践共同提升了智能体推理和训练效率，在具有挑战性的基准上使用较小模型取得了强劲结果，并为未来的智能体RL研究确立了实用基线。除这些经验见解外，我们进一步贡献了高质量、真实的端到端智能体SFT数据集和高质量RL数据集，并在包括AIME2024/AIME2025、GPQA-Diamond和LiveCodeBench-v6在内的四个挑战性基准上展示了我们的见解在提升LLMs智能体推理能力方面的有效性。使用我们的方案，4B大小的模型也能取得优于32B大小模型的智能体推理性能。代码和模型：https://github.com/Gen-Verse/Open-AgentRL
英文摘要：
Recently, the emergence of agentic RL has showcased that RL could also effectively improve the agentic reasoning ability of LLMs, yet the key design principles and optimal practices remain unclear. In this work, we conduct a comprehensive and systematic investigation to demystify reinforcement learning in agentic reasoning from three key perspectives: data, algorithm, and reasoning mode. We highlight our key insights: (i) Replacing stitched synthetic trajectories with real end-to-end tool-use trajectories yields a far stronger SFT initialization; high-diversity, model-aware datasets sustain exploration and markedly improve RL performance. (ii) Exploration-friendly techniques are crucial for agentic RL, such as clip higher, overlong reward shaping, and maintaining adequate policy entropy could improve the training efficiency. (iii) A deliberative strategy with fewer tool calls outperforms frequent tool calls or verbose self-reasoning, improving tool efficiency and final accuracy. Together, these simple practices consistently enhance agentic reasoning and training efficiency, achieving strong results on challenging benchmarks with smaller models, and establishing a practical baseline for future agentic RL research. Beyond these empirical insights, we further contribute a high-quality, real end-to-end agentic SFT dataset along with a high-quality RL dataset, and demonstrate the effectiveness of our insights in boosting the agentic reasoning ability of LLMs across four challenging benchmarks, including AIME2024/AIME2025, GPQA-Diamond, and LiveCodeBench-v6. With our recipes, 4B-sized models could also achieve superior agentic reasoning performance compared to 32B-sized models. Code and models: https://github.com/Gen-Verse/Open-AgentRL
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
