arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.22174cs.CLcs.AIcs.LG
收藏

Mosaic:多会话智能体强化学习

Mosaic: Multi-Session Agentic Reinforcement Learning

LARK, HKUST(GZ)(香港科技大学(广州)LARK实验室);Foundation Model Lab, Huawei(华为基础模型实验室);City University of Hong Kong(香港城市大学);HKUST(香港科技大学)

Yilong Xu, Minrui Xu, Xinyu Zhou, Xiang Long, Haochen Tan, Zhijiang Guo

AI总结:

针对现有智能体强化学习单会话假设与跨会话轨迹不匹配的问题,提出无评论家分层优化的Mosaic方法,在BrowseComp数据集上验证其在两类场景中性能更优且会话使用更少。

摘要:

现有大多数智能体强化学习方法假设轨迹为单会话,交互在一个持续累积的上下文内进行。然而,上下文压缩、子智能体分叉等智能体技术可能产生跨多个会话的轨迹,导致单会话训练假设与推理时执行之间存在不匹配。因此,我们引入多会话智能体强化学习,这是一种更通用的设置,支持轨迹内任意数量的会话。在该设置下,广泛采用的方法是将轨迹级优势广播到所有会话,但该方法无法区分不同会话的贡献,也无法指导轨迹内的会话分配。为解决此问题,我们提出Mosaic,一种无评论家的分层优化多会话智能体强化学习方法。在轨迹级,它将会话使用信号纳入优势估计;在会话级,它根据估计的边际贡献塑造每个会话的优势。在BrowseComp数据集上使用Qwen3.5-4B模型的实验表明,Mosaic在上下文压缩和子智能体分叉场景中均优于广播方法,同时平均使用更少的会话。进一步分析显示,我们的方法在会话效用和分配方面取得了改进,这被认为是多会话智能体强化学习未来进展的有前景目标。

英文摘要:

Most existing agentic reinforcement learning methods assume single-session trajectories, where interaction proceeds within one continuously accumulated context. However, agentic techniques such as context compaction and subagent forking may produce trajectories spanning multiple sessions, creating a mismatch between the single-session training assumption and inference-time execution. Therefore, we introduce Multi-Session Agentic Reinforcement Learning, a more general setting that supports an arbitrary number of sessions within a trajectory. Under this setting, a widely adopted method is to broadcast trajectory-level advantages to all sessions, but this does not distinguish contributions from various sessions or guide session allocation within trajectories. To address this issue, we propose Mosaic, a multi-session agentic RL method with critic-free hierarchical optimization. At the trajectory level, it incorporates session-usage signals into advantage estimation. At the session level, it shapes each session's advantage according to the estimated marginal contribution. Experiments on BrowseComp with Qwen3.5-4B show that Mosaic outperforms broadcasting in both context compaction and subagent forking scenarios while using fewer sessions on average. Further analyses show that our method achieves improvements in session utility and allocation, which we suggested is a promising target for future advances in multi-session agentic RL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑