The rapid growth of large language models (LLMs) with diverse capabilities, costs, and domains has created a critical need for intelligent model selection at inference time. While smaller models suffice for routine queries, complex tasks demand more capable models. However, static model deployment does not account for the complexity and domain of incoming queries, leading to suboptimal performance and increased costs. Dynamic routing systems that adaptively select models based on query characteristics have emerged as a solution to this challenge.
This survey provides a systematic analysis of multi-LLM routing and cascading approaches, focusing on systems that route queries across a pool of independently trained LLMs at inference time. We cover diverse routing paradigms, including query difficulty, human preferences, clustering, uncertainty quantification, reinforcement learning, multimodality, and cascading. For each paradigm, we analyze representative methods and examine key trade-offs. Beyond taxonomy, we introduce a conceptual framework that characterizes routing systems along three dimensions: when decisions are made, what information is used, and how they are computed. This perspective highlights that practical systems are often compositional, integrating multiple paradigms under operational constraints.
Our analysis demonstrates that effective multi-LLM routing requires balancing competing objectives. Choosing the optimal routing strategy depends on deployment and computational constraints. Well-designed routing systems can outperform even the most powerful individual models by strategically leveraging specialized capabilities across models while maximizing efficiency gains. Meanwhile, open challenges remain in developing and evaluating routing mechanisms that generalize across diverse architectures, modalities, and applications.
CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing
The University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Intel(英特尔);Carnegie Mellon University(卡内基梅隆大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·拉希德人工智能大学);Rutgers University(罗格斯大学)
AI Lab, Princeton University(普林斯顿大学人工智能实验室);University of Michigan(密歇根大学);Institute for Interdisciplinary Information Sciences (IIIS), Tsinghua University(清华大学交叉信息学院);Shanghai Jiao Tong University(上海交通大学);University of Edinburgh(爱丁堡大学);King’s College London(伦敦国王学院)
Informed Routing in LLMs: Smarter Token-Level Computation for Faster Inference
Institute of Digital Twin, Eastern Institute of Technology(数字孪生研究所、东部技术研究院);University of Science and Technology of China(中国科学技术大学);The Hong Kong University of Science and Technology(香港科技大学)
UniScale: Adaptive Unified Inference Scaling via Online Joint Optimization of Model Routing and Test-Time Scaling
School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)大数据研究院);School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院);College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院);Department of Computer Science and Engineering, Chinese University of Hong Kong(香港中文大学计算机科学与工程系)
Signed Rescue Routing: Harm-Aware Cascades for Efficient LLM Inference
School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院);School of Information Science and Engineering, Chongqing Jiaotong University(重庆交通大学信息科学与工程学院);School of Computer Science and Technology, Dalian University of Technology(大连理工大学计算机科学与技术学院);School of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院)
Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机学院,北京大学);LLM-Core Xiaomi(小米LLM-Core)
HierRouter: Coordinated Routing of Specialized Large Language Models via Reinforcement Learning
David S. Hippocampus Department of Computer Science Cranberry-Lemon University(Cranberry-Lemon大学计算机科学系);University of Southern California(南加州大学);DEVCOM Army Research Office(陆军研究办公室)
Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing
School of Computer Science, Peking University(北京大学计算机科学学院);MOE Key Laboratory of Computational Linguistics, Peking University(北京大学教育部计算语言学重点实验室);National Key Laboratory for Multimedia Information Processing, Peking University(北京大学国家多媒体信息处理重点实验室)
Token Level Routing Inference System for Edge Devices
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学(MBZUAI));University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Rutgers University(罗格斯大学)
A rigor-matched audit of periodic-step layer skipping for efficient llm inference: conflayers versus swift, with a supplemental analysis of trained routing alternatives
Mitigating LLM Over-Refusal via Dynamic Semantic Routing Calibration
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);CSIRO(澳大利亚联邦科学与工业研究组织);Center of Excellence for Generative AI, King Abdullah University of Science and Technology(阿卜杜拉国王科技大学生成式人工智能卓越中心)
R2E-VID: Two-Stage Robust Routing via Temporal Gating for Elastic Edge-Cloud Video Inference
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Peng Cheng Laboratory(鹏城实验室);National University of Singapore(新加坡国立大学);Beijing Kuaishou Technology Co., Ltd.(北京快手科技有限公司)
Joint Optimization of DNN Model Caching and Request Routing in Mobile Edge Computing
School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程系)