在大型语言模型评估中量化构念效度
Quantifying construct validity in large language model evaluations
AI总结：
本文提出结构能力模型，通过结合缩放定律和潜在因子模型，提升LLM评估中构念效度的量化能力。
摘要：
LLM社区常常将基准测试结果视为一般模型能力的等同物。然而，基准测试可能会有扭曲性能的问题，例如测试集污染和标注者错误。我们如何知道一个基准测试是可靠的能力指示器？这个问题涉及LLM基准测试的构念效度，要求在建模和预测LLM性能时将基准测试结果与能力区分开来。社会科学家和计算机科学家提出了形式模型——潜在因子模型和缩放定律——用于识别基准测试分数背后的潜在能力。然而，这两种技术都不足以满足构念效度的要求。潜在因子模型忽略了缩放定律，因此它们提取的能力往往代理模型大小。缩放定律忽略了测量误差，因此它们提取的能力既不可解释又过度拟合于观察到的基准测试。本论文提出了结构能力模型，这是首个从大量LLM基准测试结果中提取可解释和可推广的能力的模型。我在此模型及其两个替代模型上拟合了一个大规模样本，来自OpenLLM Leaderboard的结果。结构能力在简洁拟合指数上优于潜在因子模型，并在出分布基准预测方面优于缩放定律。这些改进的可能性在于，现有方法在适当的方式上没有将模型规模与能力区分开来。模型规模应影响能力，如同缩放定律，而这些能力应影响观察到的结果，直到测量误差，如同潜在因子模型。在结合这两种见解后，结构能力在量化LLM评估中的构念效度方面表现出更好的解释和预测能力。
英文摘要：
The LLM community often reports benchmark results as if they are synonymous with general model capabilities. However, benchmarks can have problems that distort performance, like test set contamination and annotator error. How can we know that a benchmark is a reliable indicator of some capability that we want to measure? This question concerns the construct validity of LLM benchmarks, and it requires separating benchmark results from capabilities when we model and predict LLM performance. Both social scientists and computer scientists propose formal models - latent factor models and scaling laws - for identifying the capabilities underlying benchmark scores. However, neither technique is satisfactory for construct validity. Latent factor models ignore scaling laws, and as a result, the capabilities they extract often proxy model size. Scaling laws ignore measurement error, and as a result, the capabilities they extract are both uninterpretable and overfit to the observed benchmarks. This thesis presents the structured capabilities model, the first model to extract interpretable and generalisable capabilities from a large collection of LLM benchmark results. I fit this model and its two alternatives on a large sample of results from the OpenLLM Leaderboard. Structured capabilities outperform latent factor models on parsimonious fit indices, and exhibit better out-of-distribution benchmark prediction than scaling laws. These improvements are possible because neither existing approach separates model scale from capabilities in the appropriate way. Model scale should inform capabilities, as in scaling laws, and these capabilities should inform observed results up to measurement error, as in latent factor models. In combining these two insights, structured capabilities demonstrate better explanatory and predictive power for quantifying construct validity in LLM evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
