arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.06554cs.AI
收藏

SeeUPO:具有收敛保证的序列级代理强化学习

SeeUPO: Sequence-Level Agentic-RL with Convergence Guarantees

Tongyi Lab(通义实验室);Alibaba Group(阿里巴巴集团)

Tianyi Hu, Qingxu Fu, Yanxi Chen, Zhaoyang Liu, Bolin Ding

AI总结:

SeeUPO通过逆向归纳实现多轮交互的单调改进和收敛,显著提升训练稳定性与性能

摘要:

强化学习(RL)已成为训练基于大语言模型(LLM)的AI代理的主要范式。然而,现有基础RL算法在代理场景中缺乏经过验证的收敛保证,尤其是在多轮设置中,这可能导致训练不稳定和无法收敛到最优策略。本文系统分析了不同策略更新机制和优势估计方法的组合如何影响单轮/多轮场景的收敛特性。我们发现,REINFORCE与组相对优势估计(GRAE)可以在无折扣条件下收敛到全局最优,但PPO与GRAE的组合会破坏PPO原有的单调改进性质。此外,我们证明主流基础RL算法在多轮场景中无法同时实现无 critic 和收敛保证。为了解决这一问题,我们提出了SeeUPO(序列级顺序更新策略优化),一种无 critic 且具有收敛保证的多轮交互方法。SeeUPO将多轮交互建模为依次执行的多智能体老虎机问题。通过逆向执行顺序的逐轮顺序策略更新,它通过逆向归纳确保单调改进和收敛到全局最优解。在AppWorld和BFCL v4上的实验表明,SeeUPO在现有基础算法上实现了显著的改进:在Qwen3-14B上相对提升43.3%-54.6%,在Qwen2.5-14B上相对提升24.1%-41.9%(跨基准平均),并具有更优越的训练稳定性。

英文摘要:

Reinforcement learning (RL) has emerged as the predominant paradigm for training large language model (LLM)-based AI agents. However, existing backbone RL algorithms lack verified convergence guarantees in agentic scenarios, especially in multi-turn settings, which can lead to training instability and failure to converge to optimal policies. In this paper, we systematically analyze how different combinations of policy update mechanisms and advantage estimation methods affect convergence properties in single/multi-turn scenarios. We find that REINFORCE with Group Relative Advantage Estimation (GRAE) can converge to the globally optimal under undiscounted conditions, but the combination of PPO & GRAE breaks PPO's original monotonic improvement property. Furthermore, we demonstrate that mainstream backbone RL algorithms cannot simultaneously achieve both critic-free and convergence guarantees in multi-turn scenarios. To address this, we propose SeeUPO (Sequence-level Sequential Update Policy Optimization), a critic-free approach with convergence guarantees for multi-turn interactions. SeeUPO models multi-turn interaction as sequentially executed multi-agent bandit problems. Through turn-by-turn sequential policy updates in reverse execution order, it ensures monotonic improvement and convergence to global optimal solution via backward induction. Experiments on AppWorld and BFCL v4 demonstrate SeeUPO's substantial improvements over existing backbone algorithms: relative gains of 43.3%-54.6% on Qwen3-14B and 24.1%-41.9% on Qwen2.5-14B (averaged across benchmarks), along with superior training stability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑