arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.04051cs.AI
收藏

面向大语言模型数据高效评估的统一框架

Toward a unified framework for data-efficient evaluation of large language models

Lele Liao, Qile Zhang, Ruofan Wu, Guanhua Fang

AI总结:

针对大语言模型评估成本高、现有IRT方法局限于二元指标和单一基准的问题,提出支持二元/连续指标、利用结构知识的LEGO-IRT框架,仅用3%评估题目即可实现稳定估计,误差降低最高10%,且与人类偏好更契合。

摘要:

在综合基准上评估大语言模型(LLM)是其发展的基石,但这一过程往往在计算和成本上令人难以承受。项目反应理论(IRT)通过将模型能力与题目难度解耦,为数据高效评估提供了一条颇具前景的路径,但现有基于IRT的方法存在显著局限:它们通常仅局限于二元正确性指标,无法原生处理生成式任务中使用的连续分数;且仅在单一基准上运行,忽略了不同指标或基准间的相关性等宝贵结构知识。为克服这些挑战,我们提出LEGO-IRT——一个用于数据高效LLM评估的统一且灵活的框架。LEGO-IRT的新颖设计原生支持二元和连续两种评估指标;此外,它引入了一种分解架构,将模型能力估计拆解为通用组件和特定结构(如按指标或按基准)组件,以显式建模并利用结构知识。通过在5个基准上涉及70个LLM的大量实验,我们表明LEGO-IRT仅使用总评估题目的3%就能获得稳定的能力估计;纳入结构知识可将估计误差降低最多10%,且我们的框架估计出的潜在能力可能与人类偏好更契合。

英文摘要:

Evaluating large language models (LLMs) on comprehensive benchmarks is a cornerstone of their development, yet it's often computationally and financially prohibitive. While Item Response Theory (IRT) offers a promising path toward data-efficient evaluation by disentangling model capability from item difficulty, existing IRT-based methods are hampered by significant limitations. They are typically restricted to binary correctness metrics, failing to natively handle the continuous scores used in generative tasks, and they operate on single benchmarks, ignoring valuable structural knowledge like correlations across different metrics or benchmarks. To overcome these challenges, we introduce LEGO-IRT, a unified and flexible framework for data-efficient LLM evaluation. LEGO-IRT's novel design natively supports both binary and continuous evaluation metrics. Moreover, it introduces a factorized architecture to explicitly model and leverage structural knowledge, decomposing model ability estimates into a general component and structure-specific (e.g., per-metric or per-benchmark) components. Through extensive experiments involving $70$ LLMs across $5$ benchmarks, we show that LEGO-IRT achieves stable capability estimates using just $3\%$ of the total evaluation items. We demonstrate that incorporating structural knowledge reduces estimation error by up to $10\%$ and reveal that the latent abilities estimated by our framework may align more closely with human preferences.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑