LLM routing has become a popular approach to improve the cost-quality trade-off of LLM services by adaptively selecting a model for each query. Recent work has explored a broad range of routing methods, including clustering-based routers, learned classifiers, pairwise ranking, and confidence-based approaches. Our extensive study of 21 routing methods across five benchmarks reveals a consistent phenomenon that we call the routing plateau (Fig. 1): many methods, including kNN, achieve very similar accuracy and converge to a narrow performance range that remains far below the oracle router. Our analysis supports a correctness-prediction bottleneck hypothesis: current routers primarily learn global-average model performance trends rather than fine-grained, query-specific routing signals. As a result, they collectively fail on queries that require instance-specific routing decisions. Moreover, to understand whether the plateau can be alleviated with a better training setup, we construct a 300K-query benchmark (Nine-by-300k). More data, larger encoders, and end-to-end fine-tuning improve eight routers by 1.24 pp on average, but leave the plateau largely intact. These findings suggest that further progress may require inputs beyond the query itself, such as partial output trajectories that reveal how models attempt the task.
RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing
School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);SinapisAI
VL-RouterBench: A Benchmark for Vision-Language Model Routing
Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University(1 图像处理与模式识别研究所,上海交通大学);The Hong Kong University of Science and Technology (Guangzhou)(2 香港科学与技术大学(广州));The Hong Kong University of Science and Technology(3 香港科学与技术大学)
TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing
Gradient;Soochow University(苏州大学);Independent Researcher(独立研究者);University of Southern California(南加州大学);Rice University(Rice大学);Carnegie Mellon University(卡内基梅隆大学);Shanghai Jiao Tong University(上海交通大学);University of California, Berkeley(加州大学伯克利分校);University of the Chinese Academy of Sciences(中国科学院大学);University of California, Los Angeles(加州大学洛杉矶分校)
Department of Statistics, University of Michigan(密歇根大学统计学系);IBM Research(IBM研究院);MIT-IBM Watson AI Lab(麻省理工-IBM沃森人工智能实验室);Federal University of Minas Gerais(巴西联邦大学矿务分校);Department of Statistics and Actuarial Science, University of Waterloo(滑铁卢大学统计学与精算科学系)
When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry
Institute of Science Tokyo(东京科学大学);Nanyang Technological University(南洋理工大学);Tohoku University(东北大学);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);Shandong University(山东大学)
Meta-Router: Bridging Gold-standard and Preference-based Evaluations in Large Language Model Routing
Department of Statistics, Indiana University Bloomington(印第安纳大学布卢明顿分校统计系);Department of Statistics, North Carolina State University(北卡罗来纳州立大学统计系);Department of Biostatistics, The University of Texas MD Anderson Cancer Center(德克萨斯大学MD安德森癌症中心生物统计学系)
GRIP: Algorithm-Agnostic Machine Unlearning for Mixture-of-Experts via Geometric Router Constraints
School of Computer Science(计算机科学学院);Georgia Institute of Technology(佐治亚理工学院);Department of Electrical Engineering(电气工程系);Tsinghua University(清华大学);School of Electrical and Computer Engineering(电气与计算机工程学院)
RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models
Southern University of Science and Technology(南方科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学)
Route to Rome Attack: Directing LLM Routers to Expensive Models via Adversarial Suffix Optimization
Key Laboratory of Symbolic Computation and Knowledge Engineering, MoE, Jilin University(符号计算与知识工程重点实验室,MoE,吉林大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
Department of Computer Science and Engineering, POSTECH, South Korea(韩国POSTECH计算机科学与工程系);Graduate School of Artificial Intelligence, POSTECH, South Korea(韩国POSTECH人工智能研究生院);Microsoft Research Asia, Beijing, China(中国北京微软亚洲研究院)
MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation
Shanghai Jiao Tong University(上海交通大学);GAIR;Alibaba Group(阿里巴巴集团);University of Science and Technology of China(中国科学技术大学);Shanghai Innovation Institute(上海创新研究院)
T-Router: Learning Thalamic Routing for Reasoning with Parameter-Efficient Reinforcement Learning
College of Artificial Intelligence, Tsinghua University(清华大学人工智能学院);State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院通用人工智能国家重点实验室);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院)
Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机学院,北京大学);LLM-Core Xiaomi(小米LLM-Core)