Learning Compact Representations of LLM Abilities via Item Response Theory
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);Department of Psychology, University of Southern California(美国南加州大学心理学系)
Jianhao Chen, Chenxu Wang, Gengrui Zhang, Peng Ye, Lei Bai, Wei Hu, Yuzhong Qu, Shuyue Hu
Recent years have witnessed a surge in the number of large language models (LLMs), yet efficiently managing and utilizing these vast resources remains a significant challenge. In this work, we explore how to learn compact representations of LLM abilities that can facilitate downstream tasks, such as model routing and performance prediction on new benchmarks. We frame this problem as estimating the probability that a given model will correctly answer a specific query. Inspired by the item response theory (IRT) in psychometrics, we model this probability as a function of three key factors: (i) the model's multi-skill ability vector, (2) the query's discrimination vector that separates models of differing skills, and (3) the query's difficulty scalar. To learn these parameters jointly, we introduce a Mixture-of-Experts (MoE) network that couples model- and query-level embeddings. Extensive experiments demonstrate that our approach leads to state-of-the-art performance in both model routing and benchmark accuracy prediction. Moreover, analysis validates that the learned parameters encode meaningful, interpretable information about model capabilities and query characteristics.
Beyond Scores: Diagnostic LLM Evaluation via Fine-Grained Abilities
College of Computer Science;Technology, National University of Defense Technology, Changsha, Hunan, China;State Key Laboratory of Complex \& Critical Software Environment, Changsha, Hunan, China;National Key Laboratory of Parallel;School of Humanities;Social Sciences, School of Public Administration, Beihang University, Beijing, China
IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(人工智能研究院,合肥综合性国家科学中心);School of Computing, National University of Singapore(计算学院,新加坡国立大学);NetEase Fuxi AI Lab(网易伏羲AI实验室)
An Interpretable and Scalable Framework for Evaluating Large Language Models
Department of Statistics, University of California, Riverside(加州大学河滨分校统计学系);School of Statistics and Data Science, Southeast University(东南大学统计与数据科学学院);Department of Statistics and Data Science, Southern University of Science and Technology(南方科技大学统计与数据科学系)
Bridge the Domains: Large Language Models Enhanced Cross-domain Sequential Recommendation
Xi'an Jiaotong University \& City University of Hong Kong Xi'an China;City University of Hong Kong Hong Kong China;Jilin University Changchun China;Tsinghua University Beijing China;Nanyang Technological University Singapore Singapore;Independent Researcher Beijing China;Xi'an Jiaotong University Xi'an China;Xi'an Jiaotong University \& City University of Hong Kong;City University of Hong Kong;Jilin University;Tsinghua University;Nanyang Technological University;Independent Researcher;Xi'an Jiaotong University