Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets. To address this challenge, we develop SELECT-LLM, the first framework for active model selection of LLMs. SELECT-LLM aims to find a small set of queries whose annotations are most informative for identifying the best LLM for a given task. To this end, we introduce a query selection rule based on expected information gain, computed from pairwise similarities between candidate model outputs. Because this rule only uses generated model responses, SELECT-LLM can be applied across candidate models without assumptions about their architecture or access to model weights. This makes it suitable for both open-weight and black-box LLMs. We evaluate SELECT-LLM across 23 datasets, 156 evaluated models, diverse task families, and multiple text evaluation metrics. Across all experiments, SELECT-LLM improves over the strongest baseline in every setting, with annotation cost reductions up to 81.8% for best model selection and up to 84.78% for near-best model selection.
Sub-SA: Strengthen In-context Learning via Submodular Selective Annotation
School of Electrical and Electronic Engineering, Nanyang Technological University(南洋理工大学电气与电子工程学院);School of Automation, Southeast University(东南大学自动化学院);State Key Laboratory of ASIC and System, Fudan University(复旦大学专用集成电路与系统国家重点实验室)
Exploring Imbalanced Annotations for Effective In-Context Learning
Department of Statistics and Data Science, Southern University of Science and Technology(南方科技大学统计与数据科学系);School of Economics and Finance, Xi’an Jiaotong University(西安交通大学经济金融学院);School of Mathematics and Statistics, Xi’an Jiaotong University(西安交通大学数学与统计学学院)
Prompt Selection Matters: Enhancing Text Annotations for Social Sciences with Large Language Models
Université Paris 1 Panthéon-Sorbonne(巴黎第一大学先贤祠-索邦大学);Université Paris-Saclay(巴黎萨克雷大学);Institut Jean Nicod(让·尼科德研究所);Ecole Normale Supérieure(巴黎高等师范学院);PSL-EHESS-CNRS(巴黎文理研究大学-法国高等社会科学研究院-法国国家科学研究中心)
From Random to Informed Data Selection: A Diversity-Based Approach to Optimize Human Annotation and Few-Shot Learning
Escola Politécnica, Universidade de São Paulo(圣保罗大学理工学院);Télécom Paris, Institut Polytechnique de Paris(巴黎高等电信学校,巴黎综合理工学院);Faculty of Economics, University of Porto(波尔图大学经济学院);INESC TEC(葡萄牙系统工程与计算机技术研究所)
Evaluating LLMs Without Oracle Feedback: Agentic Annotation Evaluation Through Unsupervised Consistency Signals
Australian Artificial Intelligence Institute (AAII), University of Technology Sydney, Australia(澳大利亚人工智能研究所(AAII)、悉尼技术大学);Centre for Frontier AI Research, Institute of High Performance Computing(前沿人工智能研究中心、高性能计算研究所);Agency for Science, Technology and Research, Singapore(科技研究局(新加坡));College of Computing and Data Science, Nanyang Technological University, Singapore(计算与数据科学学院、南洋理工大学)
The Harder The Better: Maintaining Supervised Fine-tuning Generalization with Less but Harder Data
Beijing Wenge Technology Co.,Ltd(北京文格科技有限公司);College of Intelligence and Computing(智能与计算学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)