arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.09418cs.CLcs.LG
收藏

大型语言模型的主动模型选择

Large Language Model Selection with Limited Annotations

TU Delft(代尔夫特理工大学);ETH Zurich(苏黎世联邦理工学院)

Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve Gürel

AI总结:

提出LLM SELECTOR框架,通过主动选择最具信息量的查询进行标注,并利用评判者预言机降低标注成本,在151个LLM和6个基准上实现高达59.62%的成本削减。

摘要:

我们提出了LLM SELECTOR,这是第一个用于大型语言模型(LLM)主动模型选择的框架。与以往依赖完全标注数据集的评估和基准测试方法不同,LLM SELECTOR在有限标注下高效识别最佳LLM。具体而言,对于任何给定任务,LLM SELECTOR自适应地选择一小部分查询进行标注,这些查询对于确定该任务的最佳模型最具信息量。为进一步降低标注成本,我们利用基于评判者的预言机标注模型。通过在包含151个LLM的6个基准上的广泛实验,我们表明LLM SELECTOR在为任务选择最佳和近最佳LLM时,将标注成本降低了高达59.62%。

英文摘要:

Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets. To address this challenge, we develop SELECT-LLM, the first framework for active model selection of LLMs. SELECT-LLM aims to find a small set of queries whose annotations are most informative for identifying the best LLM for a given task. To this end, we introduce a query selection rule based on expected information gain, computed from pairwise similarities between candidate model outputs. Because this rule only uses generated model responses, SELECT-LLM can be applied across candidate models without assumptions about their architecture or access to model weights. This makes it suitable for both open-weight and black-box LLMs. We evaluate SELECT-LLM across 23 datasets, 156 evaluated models, diverse task families, and multiple text evaluation metrics. Across all experiments, SELECT-LLM improves over the strongest baseline in every setting, with annotation cost reductions up to 81.8% for best model selection and up to 84.78% for near-best model selection.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑