arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.17200cs.CLcs.AIstat.AP
收藏

利用心理测量学方法改进LLM排行榜

Improving LLM Leaderboards with Psychometrical Methodology

Johannes Gutenberg University Mainz(美因茨约翰内斯古滕贝格大学);Goethe University Frankfurt(法兰克福大学)

Denis Federiakin

AI总结:

本文提出应用现代心理测量学方法改进LLM排行榜排名,通过Hugging Face Leaderboard数据对比证明该方法比传统平均分聚合更稳健、更有意义。

摘要:

大型语言模型(LLMs)的快速发展促使了评估其性能的基准测试的创建。这些基准类似于人类测试和调查,因为它们由旨在衡量这些系统认知行为中涌现特性的问题集组成。然而,与社会科学中研究的定义明确的特质和能力不同,这些基准所衡量的特性通常更加模糊且定义不够严格。为方便起见,最著名的基准通常被归入排行榜,汇总性能指标并实现模型间的比较。遗憾的是,这些排行榜通常依赖简单的聚合方法,例如取各基准的平均分。在本文中,我们展示了应用最初为人类测试和调查开发的现代心理测量学方法来改进LLM在排行榜上排名的优势。以Hugging Face Leaderboard的数据为例,我们将传统朴素排名方法的结果与基于心理测量学的排名进行了比较。研究结果突出了采用心理测量技术对LLM性能进行更稳健、更有意义评估的益处。

英文摘要:

The rapid development of large language models (LLMs) has necessitated the creation of benchmarks to evaluate their performance. These benchmarks resemble human tests and surveys, as they consist of sets of questions designed to measure emergent properties in the cognitive behavior of these systems. However, unlike the well-defined traits and abilities studied in social sciences, the properties measured by these benchmarks are often vaguer and less rigorously defined. The most prominent benchmarks are often grouped into leaderboards for convenience, aggregating performance metrics and enabling comparisons between models. Unfortunately, these leaderboards typically rely on simplistic aggregation methods, such as taking the average score across benchmarks. In this paper, we demonstrate the advantages of applying contemporary psychometric methodologies - originally developed for human tests and surveys - to improve the ranking of large language models on leaderboards. Using data from the Hugging Face Leaderboard as an example, we compare the results of the conventional naive ranking approach with a psychometrically informed ranking. The findings highlight the benefits of adopting psychometric techniques for more robust and meaningful evaluation of LLM performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑