Training LLMs for Multi-Step Tool Orchestration with Constrained Data Synthesis and Graduated Rewards
Amazon Services Inc.(亚马逊服务公司);Department of Computer Science and Engineering, HKUST(香港科技大学计算机科学与工程系);School of Computational Science and Engineering, Georgia Institute of Technology(佐治亚理工学院计算科学与工程学院)
Multi-step tool orchestration remains challenging for LLMs, as state-of-the-art models frequently fail on full sequence execution due to parameter errors. Training for these workflows faces two obstacles: the lack of environments supporting complex real-world API dependencies, and sparse binary rewards that provide no signal for partial correctness. We propose a reinforcement learning framework addressing both challenges. First, we construct a deterministic environment backed by a large-scale cache of real API responses, enabling constrained synthesis of valid multi-step traces with controllable complexity. Second, we introduce a graduated reward that decomposes correctness into atomic validity (call-level correctness at increasing granularity) and orchestration consistency (correct sequencing with dependency respect). On ComplexFuncBench, our approach substantially improves turn accuracy, with ablations confirming both reward components are essential. Cross-benchmark evaluation on BFCL v4 shows that the learned orchestration skills transfer to entirely different API ecosystems (e.g., agentic web search and memory management), yielding consistent gains while maintaining stable single-step performance. Code is available at https://github.com/horizon-rl/ToolOrchestrationReward
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);The Chinese University of Hong Kong Shenzhen(香港中文大学(深圳));Beijing Academy of Artificial Intelligence(北京智源人工智能研究院)
MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology(哈尔滨工业大学社会计算与交互机器人研究中心);Beijing Academy of Artificial Intelligence(北京人工智能研究院)
Department of Applied Mathematics and Theoretical Physics, University of Cambridge(应用数学与理论物理系,剑桥大学);Centre for Human-Inspired Artificial Intelligence, University of Cambridge(启发式人工智能中心,剑桥大学);African Institute for Mathematical Sciences, South Africa(南非数学科学研究所);Department of Engineering Science, University of Oxford(工程科学系,牛津大学)
Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill
Qwen Large Model Application Team, Alibaba(通义千问大模型应用团队,阿里巴巴);Sun Yat-sen University(中山大学);The Chinese University of Hong Kong(香港中文大学);Peking University(北京大学);ETH Zürich University of Zurich(苏黎世联邦理工学院)
A-MADiff: Attention-Guided Multi-Agent DRL with Diffusion Policies for Memory-Aware Task Orchestration in Mobile AIGC Networks
School of Automation, Guangdong University of Technology(广东工业大学自动化学院);City University of Hong Kong(香港城市大学);School of Information and Communication, Guilin University of Electronic Technology(桂林电子科技大学信息与通信学院);Anhui Engineering Research Center for Agricultural Product Quality Safety Digital Intelligence, Fuyang Normal University(阜阳师范大学安徽省农产品质量安全数字智能工程研究中心);School of Physics and Electronic Engineering, Fuyang Normal University(阜阳师范大学物理与电子工程学院);University of Manitoba(曼尼托巴大学)
PlexRL: Cluster-Level Orchestration of Serviceized LLM Execution for RLVR
National University of Singapore(新加坡国立大学);Beihang University(北航);Shanghai Qiji Zhifeng Co., Ltd.(上海启智风科技有限公司);Nanyang Technological University(南洋理工大学);Shanghai Innovation Institute(上海创新研究院)
GAP: Graph-Based Agent Planning with Parallel Tool Use and Reinforcement Learning
Tsinghua University(清华大学);Huazhong University of Science and Technology(华中科技大学);National University of Singapore(新加坡国立大学);Carnegie Mellon University(卡内基梅隆大学)
Agentic reinforcement learning empowers next-generation chemical language models for molecular design and synthesis
School of Electronic and Computer Engineering, Peking University(电子与计算机工程学院,北京大学);Shenzhen Graduate School, Shenzhen, 518055, China(深圳研究生院,深圳,518055,中国);International Digital Economy Academy (IDEA)(国际数字经济学院(IDEA));Peng Cheng Laboratory, Shenzhen, 518000, China(鹏城实验室,深圳,518000,中国)
AutoSynth: Automated Workflow Optimization for High-Quality Synthetic Dataset Generation via Monte Carlo Tree Search
Shanghai Innavation Institute(上海创新研究院);Shanghai Institute of AI for Education(上海人工智能教育研究院);Department of Education Information Technology(教育信息技术部);School of Computer Science and Technology(计算机科学与技术学院)