arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.04692cs.AIcs.LG
收藏

Smoothie:无标签语言模型路由

Smoothie: Label Free Language Model Routing

Stanford University(斯坦福大学)

Neel Guha, Mayee F. Chen, Trevor Chow, Ishan S. Khare, Christopher Ré

AI总结:

针对多任务场景下LLM路由需人工标注数据的问题,提出无标签路由方法Smoothie,通过构建潜变量图模型估计样本LLM质量得分并路由,14个任务中9个识别最优模型,准确率较基线提升达10个百分点。

摘要:

大型语言模型(LLMs)越来越多地用于输入可能涉及多种不同任务的应用中。近期研究发现,LLM的选择至关重要,不同LLM可能适用于不同输入样本。因此,先前方法探索了工程师如何为每个样本选择LLM(即路由)。现有路由方法大多需在人工标注数据上训练辅助模型,而本研究探索能否进行无监督路由。我们提出Smoothie,一种受弱监督启发的无标签路由方法。给定不同LLM的输出集,Smoothie基于可观测LLM输出和未知“真实”输出的嵌入表示构建潜变量图模型。利用该模型,Smoothie估计每个样本的LLM质量得分,并将样本路由至得分最高的LLM。结果显示,Smoothie的LLM质量得分与真实模型质量相关(14个任务中9个正确识别最优模型),且路由准确率较基线提升高达10个百分点。

英文摘要:

Large language models (LLMs) are increasingly used in applications where LLM inputs may span many different tasks. Recent work has found that the choice of LLM is consequential, and different LLMs may be good for different input samples. Prior approaches have thus explored how engineers might select an LLM to use for each sample (i.e. routing). While existing routing methods mostly require training auxiliary models on human-annotated data, our work explores whether it is possible to perform unsupervised routing. We propose Smoothie, a weak supervision-inspired routing approach that requires no labeled data. Given a set of outputs from different LLMs, Smoothie constructs a latent variable graphical model over embedding representations of observable LLM outputs and unknown "true" outputs. Using this graphical model, we estimate sample-dependent quality scores for each LLM, and route each sample to the LLM with the highest corresponding score. We find that Smoothie's LLM quality-scores correlate with ground-truth model quality (correctly identifying the optimal model on 9/14 tasks), and that Smoothie outperforms baselines for routing by up to 10 points accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑