Tool-Integrated Reasoning has emerged as a key paradigm to augment Large Language Models (LLMs) with computational capabilities, yet integrating tool-use into long Chain-of-Thought (long CoT) remains underexplored, largely due to the scarcity of training data and the challenge of integrating tool-use without compromising the model's intrinsic long-chain reasoning. In this paper, we introduce DART (Discovery And Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees), a reinforcement learning framework that enables spontaneous tool-use during long CoT reasoning without human annotation. DART operates by constructing dynamic rollout trees during training to discover valid tool-use opportunities, branching out at promising positions to explore diverse tool-integrated trajectories. Subsequently, a tree-based process advantage estimation identifies and credits specific sub-trajectories where tool invocation positively contributes to the solution, effectively reinforcing these beneficial behaviors. Extensive experiments on challenging benchmarks like AIME and GPQA-Diamond demonstrate that DART significantly outperforms existing methods, successfully harmonizing tool execution with long CoT reasoning.
GRPODropout: Less is More for Online Reinforcement Learning Rollouts
Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算与智能学院);Beijing Zhongguancun Academy(北京中关村学院);XinzhuAI(芯智科技)
Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rollouts
School of Information, Renmin University of China(中国人民大学信息学院);Key Laboratory of Data Engineering and Knowledge Engineering(数据工程与知识工程重点实验室);Ant Group(蚂蚁集团)
TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs
University of Chinese Academy of Sciences(中国科学院大学);Meituan(美团);MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别国家重点实验室与智能信息处理重点实验室)
DynaWeb: Model-Based Reinforcement Learning of Web Agents
Shanghai Jiao Tong University(上海交通大学);Sichuan University(四川大学);Hong Kong University of Science and Technology(香港科学与技术大学);McGill University(麦吉尔大学);Shanghai AI Lab(上海人工智能实验室);Gradient;University of Toronto(多伦多大学);Mila - Quebec AI Institute(魁北克AI研究所)
Reinforcement Learning for Chain of Thought Compression with One-Domain-to-All Generalization
CFCS, School of Computer Science, Peking University(计算机学院,北京大学);State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机学院,北京大学);LLM-Core Xiaomi(小米LLM-Core)
Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs
School of Computer Science and Technology, Beijing Jiaotong University(计算机科学与技术学院,北京交通大学);Meituan(美团);Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence(北京交通数据挖掘与具身智能重点实验室)
Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts
Mila - Québec AI Institute, Canada(魁北克人工智能研究所);Huawei Noah's Ark Lab (Montreal Research Center), Canada(华为诺亚实验室(蒙特利尔研究中心));Canada CIFAR AI Chair(加拿大CIFAR人工智能主席)
R$^2$PO: Decoupling Rollout and Inference Policies for LLM Reasoning
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国家大学)
MICRO: Multi-Fidelity Active Search for Severe Error Discovery
Institute of Neuroinformatics, University of Zurich and ETH Zurich(苏黎世大学和苏黎世联邦理工学院神经信息学研究所);ETH AI Center(苏黎世联邦理工学院人工智能中心);Department of Computational Linguistics, University of Zurich(苏黎世大学计算语言学系)