We study the problem of routing queries to large language models (LLMs) under cost, GPU resources, and concurrency constraints. Prior per-query routing methods often fail to control batch-level cost, especially under non-uniform or adversarial batching. To address this, we propose a batch-level, resource-aware routing framework that jointly optimizes model assignment for each batch while respecting cost and model capacity limits. We further introduce a robust variant that accounts for uncertainty in predicted LLM performance, along with an offline instance allocation procedure that balances quality and throughput across multiple models. Experiments on two multi-task LLM benchmarks show that robustness improves accuracy by 1-14% over non-robust counterparts (depending on the performance estimator), batch-level routing outperforms per-query methods by up to 24% under adversarial batching, and optimized instance allocation yields additional gains of up to 3% compared to a non-optimized allocation, all while strictly controlling cost and GPU resource constraints.
ConsRoute:Consistency-Aware Adaptive Query Routing for Cloud-Edge-Device Large Language Models
Department of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术系);National Key Laboratory of Smart Farm Technologies and Systems(国家智能农业技术与系统重点实验室);Department of Computer Science and Technology, Harbin Institute of Technology (ShenZhen)(哈尔滨工业大学(深圳)计算机科学与技术系)
Faculty of Informatics(信息学系);Information Technologies, Slovak University of Technology in Bratislava(信息技术,斯洛伐克技术大学布拉迪斯拉发分校);Centre for Machine Learning, University of Southern Denmark(机器学习中心,丹麦南部大学)
Routing Should Pay for Itself: Sparse Supervision for Economical LLM Routing
School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术全国重点实验室);The Hong Kong University of Science and Technology(香港科技大学);SinapisAI
RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models
Southern University of Science and Technology(南方科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学)
Learning Query-Aware Budget-Tier Routing for Runtime Agent Memory
Nanyang Technological University(南洋理工大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Illinois Chicago(伊利诺伊大学香槟分校);Tsinghua University(清华大学);Sun Yat-sen University(中山大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州))
Beyond Top-$k$ Skill Retrieval: Diversity-Aware Skill Routing for LLM Agents
Virginia Tech(弗吉尼亚理工大学);University of Southern California(南加州大学);Adobe Research(Adobe研究院);Dolby Labs(杜比实验室);Texas A&M University(德克萨斯A&M大学);Arizona State University(亚利桑那州立大学)
RealRoute: Dynamic Query Routing System via Retrieve-then-Verify Paradigm
Technical University of Denmark(技术大学);University of Exeter(埃克塞特大学);University of Toronto(多伦多大学);Rutgers University(罗格斯大学);NJIT(新 jersey 工业技术学院);University of Georgia(佐治亚大学);Wake Forest University(威克森林大学)
IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(人工智能研究院,合肥综合性国家科学中心);School of Computing, National University of Singapore(计算学院,新加坡国立大学);NetEase Fuxi AI Lab(网易伏羲AI实验室)
CR^2: Cost-Aware Risk-Controlled Routing for Wireless Device-Edge LLM Inference
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University(合作中位网创新中心,上海交通大学);Department of Broadband Communication, Pengcheng Laboratory(宽带通信部,鹏城实验室);Future Network of Intelligent Institute (FNii), the Chinese University of Hong Kong (Shenzhen)(智能网络研究所(FNii),香港中文大学(深圳));Meta
Beyond Gemini-3-Pro: Revisiting LLM Routing and Aggregation at Scale
Shanghai AI Lab(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Fudan University(复旦大学);University of Science and Technology of China(中国科学技术大学);Northwestern Polytechnical University(西北工业大学);Northeastern University(东北大学)
Deep Active Learning: Unified and Principled Method for Query and Training
Université Laval(拉瓦尔大学);Mila, Canada CIFAR AI Chair(Mila(加拿大)CIFAR人工智能主席);University of Western Ontario(西安大略大学);Vector Institute(Vector研究所(注:该机构常见译名为“Vector研究院”))
From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing
School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);National Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);Hong Kong University of Science and Technology(香港科学与技术大学);Frontier Robotics(前沿机器人)
Yandex(Yandex(央捷科斯));Lomonosov Moscow State University(罗蒙诺索夫莫斯科国立大学);Moscow Institute of Physics and Technology(莫斯科物理技术学院);National Research University Higher School of Economics(国立研究型大学高等经济学院)