智能体熵平衡策略优化
Agentic Entropy-Balanced Policy Optimization
AI总结：
针对智能体强化学习中熵信号过度依赖导致的训练崩溃问题，提出AEPO算法，通过动态熵平衡采样和熵感知策略优化，在14个数据集上超越7种主流算法，仅用1K样本即取得显著性能提升。
摘要：
近年来，智能体强化学习（Agentic Reinforcement Learning, Agentic RL）在激励网络智能体的多轮、长时程工具使用能力方面取得了显著进展。虽然主流智能体强化学习算法在熵的引导下自主探索高不确定性的工具调用步骤，但对熵信号的过度依赖可能施加额外约束，导致训练崩溃。本文深入探讨了熵所带来的挑战，并提出了智能体熵平衡策略优化（Agentic Entropy-Balanced Policy Optimization, AEPO），一种旨在在轨迹采样和策略更新阶段平衡熵的智能体强化学习算法。AEPO包含两个核心组件：（1）动态熵平衡的轨迹采样机制，通过熵预监测自适应地分配全局和分支采样预算，同时对连续高熵工具调用步骤施加分支惩罚，以防止过度分支问题；（2）熵平衡策略优化，在高熵裁剪项中插入停止梯度操作，以保留并适当重新缩放高熵标记上的梯度，同时引入熵感知的优势估计，优先学习高不确定性标记。在14个具有挑战性的数据集上的结果表明，AEPO始终优于7种主流强化学习算法。仅使用1K条强化学习样本，配备AEPO的Qwen3-14B即取得了令人瞩目的结果：在GAIA上Pass@1为47.6%，在Humanity's Last Exam上为11.2%，在WebWalker上为43.0%；在GAIA上Pass@5为65.0%，在Humanity's Last Exam上为26.0%，在WebWalker上为70.0%。进一步分析表明，AEPO在保持稳定策略熵的同时提高了轨迹采样多样性，促进了可扩展的网络智能体训练。
英文摘要：
Recently, Agentic Reinforcement Learning (Agentic RL) has made significant progress in incentivizing the multi-turn, long-horizon tool-use capabilities of web agents. While mainstream agentic RL algorithms autonomously explore high-uncertainty tool-call steps under the guidance of entropy, excessive reliance on entropy signals can impose further constraints, leading to the training collapse. In this paper, we delve into the challenges caused by entropy and propose the Agentic Entropy-Balanced Policy Optimization (AEPO), an agentic RL algorithm designed to balance entropy in both the rollout and policy update phases. AEPO comprises two core components: (1) a dynamic entropy-balanced rollout mechanism that adaptively allocate global and branch sampling budget through entropy pre-monitoring, while imposing a branch penalty on consecutive high-entropy tool-call steps to prevent over-branching issues; and (2) Entropy-Balanced Policy Optimization that inserts a stop-gradient operation into the high-entropy clipping term to preserve and properly rescale gradients on high-entropy tokens, while incorporating entropy-aware advantage estimation to prioritize learning on high-uncertainty tokens. Results across 14 challenging datasets show that AEPO consistently outperforms 7 mainstream RL algorithms. With just 1K RL samples, Qwen3-14B with AEPO achieves impressive results: 47.6% on GAIA, 11.2% on Humanity's Last Exam, and 43.0% on WebWalker for Pass@1; 65.0% on GAIA, 26.0% on Humanity's Last Exam, and 70.0% on WebWalker for Pass@5. Further analysis reveals that AEPO improves rollout sampling diversity while maintaining stable policy entropy, facilitating scalable web agent training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
