PATCH! 基于心理测量学辅助的大型语言模型与人类群体基准测试：以八年级数学能力为例
PATCH! {P}sychometrics-{A}ssis{T}ed Ben{CH}marking of Large Language Models against Human Populations: A Case Study of Proficiency in 8th Grade Mathematics
AI总结：
针对现有LLM基准测试在测量质量、项目层面和人类参照上的不足，提出心理测量学辅助框架PATCH，实现与人类群体的有效比较，并在八年级数学上验证了其评估结果差异，同时发布4个高质量数据集。
摘要：
许多现有的大型（多模态）语言模型（LLM）基准测试侧重于衡量其学术能力，通常也关注将模型性能与人类考生进行比较。尽管此类基准已被证明对LLM的发展至关重要，但它们存在若干局限性，包括测量质量存疑（例如，它们是否以可靠的方式测量了本应测量的内容？）、缺乏项目层面的质量评估（例如，某些项目是否比其他项目更重要或更难？）以及人类群体参照不明确（例如，模型可以与哪些人进行比较？）。针对这些挑战，我们提出利用心理测量学（一门致力于测量学术能力等潜在变量的学科）的知识来辅助LLM基准测试。我们做出四项主要贡献。首先，我们反思了当前LLM基准测试的发展，并将其与基于心理测量学的测试开发进行对比。其次，我们引入了PATCH：一个用于LLM心理测量学辅助基准测试的新框架。PATCH解决了上述局限性，特别是能够实现LLM与人类群体之间的有效比较。第三，我们通过测量多个LLM在八年级数学中针对56个人类群体的能力来演示PATCH。我们表明，采用基于心理测量学的方法得出的评估结果与基于当前基准测试实践的结果存在差异。第四，我们发布了4个高质量数据集，以支持衡量和比较LLM在小学数学和科学方面与人类群体的能力。
英文摘要：
Many existing benchmarks of large (multimodal) language models (LLMs) focus on measuring LLMs' academic proficiency, often with also an interest in comparing model performance with human test takers'. While such benchmarks have proven key to the development of LLMs, they suffer from several limitations, including questionable measurement quality (e.g., Do they measure what they are supposed to in a reliable way?), lack of quality assessment on the item level (e.g., Are some items more important or difficult than others?) and unclear human population reference (e.g., To whom can the model be compared?). In response to these challenges, we propose leveraging knowledge from psychometrics -- a field dedicated to the measurement of latent variables like academic proficiency -- into LLM benchmarking. We make four primary contributions. First, we reflect on current LLM benchmark developments and contrast them with psychometrics-based test development. Second, we introduce PATCH: a novel framework for {P}sychometrics-{A}ssis{T}ed ben{CH}marking of LLMs. PATCH addresses the aforementioned limitations. In particular, PATCH enables valid comparison between LLMs and human populations. Third, we demonstrate PATCH by measuring several LLMs' proficiency in 8th grade mathematics against 56 human populations. We show that adopting a psychometrics-based approach yields evaluation outcomes that diverge from those based on current benchmarking practices. Fourth, we release 4 high-quality datasets to support measuring and comparing LLM proficiency in grade school mathematics and science with human populations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
