Advances in large language models (LLMs) are driving a shift toward using reinforcement learning (RL) to train agents from iterative, multi-turn interactions across tasks. However, multi-turn RL remains challenging as rewards are often sparse or delayed, and environments can be stochastic. In this regime, naive trajectory sampling can hinder exploitation and induce mode collapse. We propose TSR (Trajectory-Search Rollouts), a training-time approach that repurposes test-time scaling ideas for improved per-turn rollout generation. TSR performs lightweight tree-style search to construct higher-quality trajectories by selecting promising actions and trajectory prefixes during rollout generation. This improves rollout quality while preserving stable policy optimization and remains compatible with standard policy-gradient optimizers by design. Across Sokoban, FrozenLake, and WebShop, TSR achieves success-rate gains of up to 15 percentage points and converges in fewer optimization steps, while trading additional training-time rollout compute for stronger policies that require no search at inference time. By moving search from test time to the rollout stage of training, TSR provides a modular mechanism for stronger multi-turn agent learning, complementary to existing frameworks and rejection-sampling-style selection methods.
DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning
School of Computer Science \& Beijing Key Laboratory of Software;Hardware Cooperative Artificial Intelligence Systems, Peking University, Beijing, China;School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China;Institute of Computational Social Science, Peking University (Qingdao), Qingdao, China
SLIM-RL: Risk-Budgeted Random-Masking RL for Diffusion LLMs Without Trajectory Slicing
Technical University of Denmark(丹麦技术大学);MBZUAI Institute of Foundation Models(穆罕默德·本·扎耶德人工智能大学基础模型研究所);Iowa State University(爱荷华州立大学);Red Hat AI Innovation(红帽人工智能创新实验室);MIT–IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)
R$^2$PO: Decoupling Rollout and Inference Policies for LLM Reasoning
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国家大学)
StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction
The Chinese University of Hong Kong(香港中文大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Georgia(佐治亚大学);University of Oxford(牛津大学);Shenzhen Loop Area Institute(深圳河套学院)
RASFT: Rollout-Adaptive Supervised Fine-Tuning for Reasoning
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);New Jersey Institute of Technology(新泽西理工学院);Institute of Computing Technology, CAS(中国科学院计算技术研究所)
Learning with a Single Rollout via Pass@k Policy Optimization
University of Alberta(阿尔伯塔大学);BIGAI(北京通用人工智能研究院);University of Toronto(多伦多大学);McGill University, Mila(麦吉尔大学,米拉);Nvidia Research(英伟达研究院);Amii(阿尔伯塔机器智能研究所);CIFAR AI Chair(CIFAR人工智能教席)
ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward
LMU Munich(慕尼黑大学);Harvard University(哈佛大学);University of Cambridge(剑桥大学);Mina AI;Konrad Zuse School of Excellence in Reliable AI (relAI)(康拉德·楚泽可靠人工智能卓越学校(relAI))
Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation
Beijing Institute of Technology(北京理工大学);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI);DataCanvas;Beijing University of Posts and Telecommunications(北京邮电大学);Shenzhen MSU-BIT University(深圳MSU-BIT大学)
Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learning
Beihang University(北京航空航天大学);Zhongguancun Academy(中关村学院);Communication University of China(中国传媒大学);Peking University(北京大学);Hangzhou Innovation Institute of Beihang University(北京航空航天大学杭州创新研究院)
Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
South China University of Technology(华南理工大学);Tsinghua University(清华大学);The Hong Kong University of Science and Technology(香港科技大学);Huawei 2012 Laboratories(华为2012实验室);Shenzhen Technology University(深圳技术大学)
Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
Georgia Institute of Technology(佐治亚理工学院);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Manchester(曼彻斯特大学);NVIDIA;Independent(独立);Emory University(埃默里大学)