arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.00844cs.AI
收藏

通过项目反应理论学习大语言模型能力的紧凑表示

Learning Compact Representations of LLM Abilities via Item Response Theory

State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);Department of Psychology, University of Southern California(美国南加州大学心理学系)

Jianhao Chen, Chenxu Wang, Gengrui Zhang, Peng Ye, Lei Bai, Wei Hu, Yuzhong Qu, Shuyue Hu

AI总结:

针对大语言模型资源高效管理利用的难题,受项目反应理论启发提出耦合模型与查询嵌入的混合专家网络以学习LLM能力的紧凑表示,在模型路由和基准准确率预测任务上达到SOTA,且参数具备可解释性。

摘要:

近年来,大语言模型(LLM)的数量激增,但如何高效管理和利用这些庞大的资源仍是一项重大挑战。在本研究中,我们探索如何学习LLM能力的紧凑表示,以支撑模型路由、新基准性能预测等下游任务。我们将该问题定义为估计给定模型正确回答特定查询的概率。受心理测量学中项目反应理论(IRT)的启发,我们将这一概率建模为三个关键因素的函数:(1)模型的多技能能力向量;(2)用于区分不同技能水平模型的查询区分度向量;(3)查询的难度标量。为联合学习这些参数,我们提出了一种混合专家(MoE)网络,用于耦合模型级与查询级嵌入。大量实验表明,我们的方法在模型路由和基准准确率预测两项任务上均达到了当前最优性能。此外,分析验证了所学习的参数编码了关于模型能力和查询特征的有意义、可解释的信息。

英文摘要:

Recent years have witnessed a surge in the number of large language models (LLMs), yet efficiently managing and utilizing these vast resources remains a significant challenge. In this work, we explore how to learn compact representations of LLM abilities that can facilitate downstream tasks, such as model routing and performance prediction on new benchmarks. We frame this problem as estimating the probability that a given model will correctly answer a specific query. Inspired by the item response theory (IRT) in psychometrics, we model this probability as a function of three key factors: (i) the model's multi-skill ability vector, (2) the query's discrimination vector that separates models of differing skills, and (3) the query's difficulty scalar. To learn these parameters jointly, we introduce a Mixture-of-Experts (MoE) network that couples model- and query-level embeddings. Extensive experiments demonstrate that our approach leads to state-of-the-art performance in both model routing and benchmark accuracy prediction. Moreover, analysis validates that the learned parameters encode meaningful, interpretable information about model capabilities and query characteristics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑