衡量重要之事：大语言模型基准中的构念效度
Measuring what Matters: Construct Validity in Large Language Model Benchmarks
AI总结：
本文通过29名专家对445个LLM基准的系统综述，揭示了现有基准在构念效度上的不足，并提出八项关键建议以改进基准开发。
摘要：
评估大语言模型（LLMs）对于在部署前评估其能力以及识别安全或鲁棒性问题至关重要。可靠地测量诸如“安全性”和“鲁棒性”等抽象且复杂的现象，需要较强的构念效度，即所使用的测量能够代表对该现象而言重要的内容。我们组建了一个由29名专家评审组成的团队，对来自自然语言处理和机器学习领域顶级会议的445个LLM基准进行了系统综述。在所评审的文章中，我们发现了与被测量现象、任务和评分指标相关的模式，这些模式削弱了所得结论的有效性。为解决这些不足，我们向研究人员和从业者提供了八项关键建议以及详细且可操作的指导，以帮助他们开发LLM基准。
英文摘要：
Evaluating large language models (LLMs) is crucial for both assessing their capabilities and identifying safety or robustness issues prior to deployment. Reliably measuring abstract and complex phenomena such as 'safety' and 'robustness' requires strong construct validity, that is, having measures that represent what matters to the phenomenon. With a team of 29 expert reviewers, we conduct a systematic review of 445 LLM benchmarks from leading conferences in natural language processing and machine learning. Across the reviewed articles, we find patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims. To address these shortcomings, we provide eight key recommendations and detailed actionable guidance to researchers and practitioners in developing LLM benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
