While multi-agent systems (MAS) promise elevated intelligence through coordination of agents, current approaches to automatic MAS design under-deliver. Such shortcomings stem from two key factors: (1) methodological complexity - agent orchestration is performed using sequential, code-level execution that limits global system-level holistic reasoning and scales poorly with agent complexity - and (2) efficacy uncertainty - MAS are deployed without understanding if there are tangible benefits compared to single-agent systems (SAS). We propose MASOrchestra, a training-time framework that formulates MAS orchestration as a function-calling reinforcement learning problem with holistic orchestration, generating an entire MAS at once. In MAS-Orchestra, complex, goal-oriented subagents are abstracted as callable functions, enabling global reasoning over system structure while hiding internal execution details. To rigorously study when and why MAS are beneficial, we introduce MASBENCH, a controlled benchmark that characterizes tasks along five axes: Depth, Horizon, Breadth, Parallel, and Robustness. Our analysis reveals that MAS gains depend critically on task structure, verification protocols, and the capabilities of both orchestrator and subagents, rather than holding universally. Guided by these insights, MAS-Orchestra achieves consistent improvements on public benchmarks including mathematical reasoning, multi-hop QA, and search-based QA, while achieving more than 10x efficiency over strong baselines. Together, MAS-Orchestra and MASBENCH enable better training and understanding of MAS in the pursuit of multi-agent intelligence.
Beyond the Strongest LLM: Multi-Turn Multi-Agent Orchestration vs. Single LLMs on Benchmarks
Arizona State University(亚利桑那州立大学);University of Pennsylvania(宾夕法尼亚大学);Gradient;Nanyang Technological University(南洋理工大学);Simon Fraser University(西蒙弗雷泽大学);University of California, Santa Barbara(加利福尼亚大学圣塔芭芭拉分校);Dropbox;Google DeepMind(谷歌DeepMind)
Small Model as Master Orchestrator: Learning Unified Agent-Tool Orchestration with Parallel Subtask Decomposition
Shanghai Jiao Tong University(上海交通大学);Fudan University(复旦大学);Shanghai AI Lab(上海人工智能实验室);Sichuan University(四川大学);The Chinese University of Hong Kong(香港中文大学)
Agentic AI with Orchestrator-Agent Trust: A Modular Visual Classification Framework with Trust-Aware Orchestration and RAG-Based Reasoning
University of the Peloponnese, Department of Informatics and Telecommunications(希腊皮埃蒙特大学信息与电信系);Cornell University, Department of Biological and Environmental Engineering(康奈尔大学生物与环境工程系)
LitVISTA: A Benchmark for Narrative Orchestration in Literary Text
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);University of Science and Technology of China(中国科学技术大学);University of Melbourne(墨尔本大学)
OrchJail: Jailbreaking Tool-Calling Text-to-Image Agents by Orchestration-Guided Fuzzing
Institute of Software, Chinese Academy of Sciences, Beijing, China;Science \& Technology on Integrated Information System Laboratory, Beijing, China;State Key Laboratory of Complex System Modeling;University of Chinese Academy of Sciences, Beijing, China