arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.11767cs.AIcs.CLcs.LG
收藏

TSR:用于LLM代理多轮RL的轨迹搜索

TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agents

Technical University Munich(慕尼黑技术大学);IBM Research(IBM研究院)

Aladin Djuhera, Swanand Kadhe, Farhan Ahmed, Syed Zawad, Heiko Ludwig, Holger Boche

AI总结:

本文提出TSR,一种在训练时改进每轮轨迹生成的方法,通过轻量级树状搜索构造高质量轨迹,提升rollout质量和学习稳定性,适用于多轮RL任务。

摘要:

大规模语言模型(LLMs)的进步正在推动使用强化学习(RL)来训练代理,从跨任务的迭代、多轮交互中学习。然而,多轮RL仍然具有挑战性,因为奖励通常稀疏或延迟,而环境可能是随机的。在这种情况下,朴素的轨迹采样会阻碍利用并导致模式崩溃。我们提出了TSR(轨迹搜索rollouts),一种训练时的方法,重新利用测试时扩展的想法以改进每轮rollout生成。TSR通过基于状态的反馈在每个回合中选择高分动作,进行轻量级树状搜索来构造高质量轨迹。这提高了rollout质量并稳定了学习,同时与标准策略梯度优化器兼容,使TSR对优化器无偏见。我们用best-of-N、beam和浅层前瞻搜索实例化TSR,并与PPO和GRPO配对,在Sokoban、FrozenLake和WebShop任务中实现高达15%的性能提升和更稳定的训练,仅需适度增加一次训练计算。通过将搜索从推理时间转移到训练的rollout阶段,TSR提供了一种模块化且通用的机制,用于更强的多轮代理学习,与现有框架和拒绝采样式选择方法互补。

英文摘要:

Advances in large language models (LLMs) are driving a shift toward using reinforcement learning (RL) to train agents from iterative, multi-turn interactions across tasks. However, multi-turn RL remains challenging as rewards are often sparse or delayed, and environments can be stochastic. In this regime, naive trajectory sampling can hinder exploitation and induce mode collapse. We propose TSR (Trajectory-Search Rollouts), a training-time approach that repurposes test-time scaling ideas for improved per-turn rollout generation. TSR performs lightweight tree-style search to construct higher-quality trajectories by selecting promising actions and trajectory prefixes during rollout generation. This improves rollout quality while preserving stable policy optimization and remains compatible with standard policy-gradient optimizers by design. Across Sokoban, FrozenLake, and WebShop, TSR achieves success-rate gains of up to 15 percentage points and converges in fewer optimization steps, while trading additional training-time rollout compute for stronger policies that require no search at inference time. By moving search from test time to the rollout stage of training, TSR provides a modular mechanism for stronger multi-turn agent learning, complementary to existing frameworks and rejection-sampling-style selection methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑