SERO：面向长视距智能体强化学习的可执行专家后续学习
SERO: Learning from Executable Oracle Continuations for Long-Horizon Agentic Reinforcement Learning
AI总结：
SERO 是一种面向长视距智能体强化学习的方法，通过引入可执行专家后续与策略可达性加权，在 ALFWorld、WebShop 基准上提升了性能。
摘要：
长视距智能体强化学习受限于当前策略访问的未来状态，当 rollout 未能达成有用行为时，改进的信用分配无法为从未遇到的状态和动作提供监督。在线蒸馏（OPD）引入教师对策略诱导状态的指导，近期多轮变体优化了该指导的调度、重放、细化或验证方式，但未直接将专家控制的未来本身作为训练轨迹。我们提出**以状态为根的可执行 rollout 优化（SERO）**，为在线智能体强化学习补充可执行专家后续：SERO 恢复当前策略实际到达的状态，让仅用于训练的专家继续与真实环境交互，使下游观测和专家动作可用于学习；随后应用**策略可达性加权**，即按冻结行为策略与前序专家动作的兼容性对每个专家步骤加权，当策略与专家分支兼容性提升时，更深层的监督可生效。专家损失与基础强化学习估计器分开优化，推理时不使用专家。我们在两个具挑战性的交互智能体基准 ALFWorld 和 WebShop 上，使用 Qwen2.5-1.5B-Instruct 和 Qwen2.5-7B-Instruct 评估 SERO：在 ALFWorld 上，学习完整可执行后续仅监督根专家动作效果显著更好，而策略可达性加权进一步提升了分布外性能。
英文摘要：
Long-horizon agentic RL is constrained by the futures visited by the current policy. When rollouts fail to reach useful behaviors, improved credit assignment cannot provide supervision for states and actions that were never encountered. On-policy distillation (OPD) introduces teacher guidance on policy-induced states, and recent multi-turn variants improve how this guidance is scheduled, replayed, refined, or validated. However, they do not directly use the oracle-controlled future itself as the training trajectory. We introduce \textbf{State-Rooted Executable Rollout Optimization (SERO)}, which augments on-policy agentic RL with executable oracle continuations. SERO restores a state actually reached by the current policy and lets a training-only oracle continue interacting with the real environment, making the downstream observations and oracle actions themselves available for learning. It then applies \textbf{policy-reachability weighting}, which weights each oracle step by the frozen behavior policy's compatibility with the preceding oracle actions. This allows deeper supervision to become effective as the policy becomes more compatible with the oracle branch. The oracle loss is optimized separately from the base RL estimator and no oracle is used at inference time. We evaluate SERO on two challenging interactive-agent benchmarks, ALFWorld and WebShop, using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. On ALFWorld, learning the full executable continuation is substantially more effective than supervising only the root oracle action, while policy-reachability weighting further improves out-of-distribution performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
