Evaluating large language models (LLMs) on comprehensive benchmarks is a cornerstone of their development, yet it's often computationally and financially prohibitive. While Item Response Theory (IRT) offers a promising path toward data-efficient evaluation by disentangling model capability from item difficulty, existing IRT-based methods are hampered by significant limitations. They are typically restricted to binary correctness metrics, failing to natively handle the continuous scores used in generative tasks, and they operate on single benchmarks, ignoring valuable structural knowledge like correlations across different metrics or benchmarks. To overcome these challenges, we introduce LEGO-IRT, a unified and flexible framework for data-efficient LLM evaluation. LEGO-IRT's novel design natively supports both binary and continuous evaluation metrics. Moreover, it introduces a factorized architecture to explicitly model and leverage structural knowledge, decomposing model ability estimates into a general component and structure-specific (e.g., per-metric or per-benchmark) components. Through extensive experiments involving $70$ LLMs across $5$ benchmarks, we show that LEGO-IRT achieves stable capability estimates using just $3\%$ of the total evaluation items. We demonstrate that incorporating structural knowledge reduces estimation error by up to $10\%$ and reveal that the latent abilities estimated by our framework may align more closely with human preferences.
Vrije University of Amsterdam University of Amsterdam Centrum Wiskunde \& Informatic;University of Amsterdam National Institute of Informatics Centrum Wiskunde \& Informatic;Centrum Wiskunde \& Informatic Leiden University University College London;Vrije University of Amsterdam;Centrum Wiskunde \& Informatica Technische Universiteit Delft Guangzhou University;Vrije University of Amsterdam Centrum Wiskunde \& Informatic
LEGO-Compiler: Enhancing Neural Compilation Through Translation Composability
SKLP, Institute of Computing Technology, CAS University of Chinese Academy of Sciences, Beijing, China(SKLP,计算技术研究所,中国科学院大学,北京,中国);University of Leeds, UK(利兹大学,英国);Zhongguancun Laboratory, Beijing, China(中关村实验室,北京,中国)
Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University(同济大学上海智能自主系统研究院);College of Design and Innovation, Tongji University(同济大学设计创意学院)
Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translation
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Nanjing University(南京大学);The University of Hong Kong(香港大学);University of California, Santa Barbara(加州大学圣巴巴拉分校)
Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Reality
Centrum Wiskunde & Informatica(数学与计算机科学中心);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);University College London(伦敦大学学院);Shandong University(山东大学);TU Delft(代尔夫特理工大学)
LEGO-Prover: Neural Theorem Proving with Growing Libraries
Sun Yat-sen University(中山大学);The Chinese University of Hong Kong(香港中文大学);Huawei HiSilicon(华为海思);City University of Hong Kong(香港城市大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学)