arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.09692cs.CLcs.AIcs.LG
收藏

基于生成数据的路由:无标注LLM技能估计与专家选择

Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection

UNC Chapel Hill(北卡罗来纳大学教堂山分校);Capital One(Capital One公司);The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Tianyi Niu, Justin Chih-Yao Chen, Genta Indra Winata, Shi-Xiong Zhang, Supriyo Chakraborty, Sambit Sahu, Yue Zhang, Elias Stengel-Eskin, Mohit Bansal

AI总结:

本文提出CASCAL,一种通过共识投票和层次聚类提升LLM路由性能的查询-only路由器,在弱生成器数据下表现更优。

摘要:

大型语言模型(LLM)路由器动态选择给定输入的最佳模型。现有方法通常假设可以获取真实标注数据,但在实践中往往不可用,尤其是当用户请求分布异质且未知时。我们引入基于生成数据的路由(RGD),这是一种具有挑战性的设置,其中路由器仅在由生成器LLM生成的查询和答案上进行训练。我们评估了查询-答案路由器(使用查询和标签)和仅查询路由器,跨四个多样化的基准和12个模型,发现随着生成器质量下降,查询-答案路由器性能下降得比仅查询路由器更快。我们的分析揭示了有效生成器的两个关键特性:它们必须准确回答自己的问题,且它们的问题必须在模型池中产生足够的性能差异。我们随后展示了如何通过过滤这些特性来提高生成数据的质量。我们进一步提出了CASCAL,一种新的仅查询路由器,通过共识投票估计模型正确性,并通过层次聚类识别模型特定的技能领域。CASCAL在弱生成器数据上训练时表现更加稳健,当训练于弱生成器数据时,其绝对准确率比最佳的查询-答案路由器高出4.6%。

英文摘要:

Large Language Model (LLM) routers dynamically select optimal models for given inputs. Existing approaches typically assume access to ground-truth labeled data, which is often unavailable in practice, especially when user request distributions are heterogeneous and unknown. We introduce Routing with Generated Data (RGD), a challenging setting in which routers are trained exclusively on generated queries and answers produced from high-level task descriptions by generator LLMs. We evaluate query-answer routers (using both queries and labels) and query-only routers across four diverse benchmarks and 12 models, finding that query-answer routers degrade faster than query-only routers as generator quality decreases. Our analysis reveals two crucial characteristics of effective generators: they must accurately respond to their own questions, and their questions must produce sufficient performance differentiation among the model pool. We then show how filtering for these characteristics can improve the quality of generated data. We further propose CASCAL, a novel query-only router that estimates model correctness through consensus voting and identifies model-specific skill niches via hierarchical clustering. CASCAL is substantially more robust to generator quality, outperforming the best query-answer router by 4.6% absolute accuracy when trained on weak generator data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑