机遇并非可实现性：多大型语言模型（LLM）路由的选择有效诊断方法

中文摘要 AI 辅助

神谕路由衡量一组语言模型（LLM）通过逐查询选择可获得的增益，但该诊断存在两个缺陷：针对同一示例上选择的最佳固定模型进行测试会使配对推理失效，且全信息神谕能看到所有结果，而这是任何可部署路由器都无法观测到的。我们区分了三个估计量：结果神谕机遇（outcome-oracle opportunity）、从声明的预回答信号中获得的贝叶斯最优增益，以及学习到的路由器的保留增益；并证明了选择有效置信区间，该区间在选择最佳固定模型或路由器族的最佳成员、信号信息三明治以及从次模互补覆盖构建紧凑池的(1-1/e)贪心保证下依然有效。在六个族的八个检查点、四个基准上，选择有效区间证实，每个任务的总体神谕差距为9.7至30.7个百分点，但最强的可部署提示路由器仅能恢复其中的7.5%至14.4%，且所测试的十一个策略中最佳者的同时区间在所有任务中下限均为零。神谕机遇的可实现份额很小且可证实：强路由器优于最佳固定模型，且大部分差距仍然存在。

英文摘要

Oracle routing measures how much a pool of language models could gain from per-query selection, but the diagnostic has two flaws: testing against a best fixed model selected on the same examples invalidates paired inference, and a full-information oracle sees outcomes no deployable router observes. We separate three estimands (outcome-oracle opportunity, the Bayes-optimal gain from a declared pre-answer signal, and the held-out gain of a learned router) and prove selection-valid confidence intervals that survive choosing the best fixed model or the best member of a router family, a signal-information sandwich, and a $(1-1/e)$ greedy guarantee for building compact pools from submodular complementary coverage. On eight checkpoints from six families over four benchmarks, selection-valid intervals certify a population oracle gap of $9.7$--$30.7$ points on every task, yet the strongest deployable prompt router recovers only $7.5$--$14.4\%$ of it, and the simultaneous interval for the best of eleven tested policies has lower limit zero throughout. The realizable share of oracle opportunity is small and certifiable: strong routers beat the best fixed model, and most of the gap remains.

发表机构

• Iowa State University（爱荷华州立大学）
• BRAC University（BRAC大学）

机构由 AI 辅助整理，请以论文原文为准。