本文提出应用现代心理测量学方法改进LLM排行榜排名,通过Hugging Face Leaderboard数据对比证明该方法比传统平均分聚合更稳健、更有意义。
摘要:
大型语言模型(LLMs)的快速发展促使了评估其性能的基准测试的创建。这些基准类似于人类测试和调查,因为它们由旨在衡量这些系统认知行为中涌现特性的问题集组成。然而,与社会科学中研究的定义明确的特质和能力不同,这些基准所衡量的特性通常更加模糊且定义不够严格。为方便起见,最著名的基准通常被归入排行榜,汇总性能指标并实现模型间的比较。遗憾的是,这些排行榜通常依赖简单的聚合方法,例如取各基准的平均分。在本文中,我们展示了应用最初为人类测试和调查开发的现代心理测量学方法来改进LLM在排行榜上排名的优势。以Hugging Face Leaderboard的数据为例,我们将传统朴素排名方法的结果与基于心理测量学的排名进行了比较。研究结果突出了采用心理测量技术对LLM性能进行更稳健、更有意义评估的益处。
英文摘要:
The rapid development of large language models (LLMs) has necessitated the creation of benchmarks to evaluate their performance. These benchmarks resemble human tests and surveys, as they consist of sets of questions designed to measure emergent properties in the cognitive behavior of these systems. However, unlike the well-defined traits and abilities studied in social sciences, the properties measured by these benchmarks are often vaguer and less rigorously defined. The most prominent benchmarks are often grouped into leaderboards for convenience, aggregating performance metrics and enabling comparisons between models. Unfortunately, these leaderboards typically rely on simplistic aggregation methods, such as taking the average score across benchmarks. In this paper, we demonstrate the advantages of applying contemporary psychometric methodologies - originally developed for human tests and surveys - to improve the ranking of large language models on leaderboards. Using data from the Hugging Face Leaderboard as an example, we compare the results of the conventional naive ranking approach with a psychometrically informed ranking. The findings highlight the benefits of adopting psychometric techniques for more robust and meaningful evaluation of LLM performance.
La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America
SomosNLP;ETSIT, Universidad Politécnica de Madrid(ETSIT,西班牙马德里理工大学);Barcelona Supercomputing Center(巴塞罗那超级计算中心);Hugging Face;Universidad Carlos III de Madrid(马德里卡洛斯三世大学);Instituto de Ingeniería del Conocimiento(知识工程研究所);Centro HiTZ - Ixa, Universidad del País Vasco UPV/EHU(HiTZ-Ixa中心,巴斯克大学);LIACS, Leiden University(LIACS,莱顿大学);Universidad de Jaén(杰嫩大学);Universidad Nacional de Córdoba(科尔多瓦国立大学);Universidad Nacional Autónoma de México(墨西哥国立自治大学);Universidad de la República, Uruguay(乌拉圭共和国大学)
La Leaderboard是一个开源项目,旨在评估西班牙及拉丁美洲各种语言和方言的生成式大型语言模型,通过社区驱动的方式促进多样化语言模型的发展。
Language Imbalance Driven Rewarding for Multilingual Self-improving
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Wuhan AI Research(武汉人工智能研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-Text Rationales
University of Southern California(南加州大学);Tsinghua University(清华大学);Meta AI;Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学)
Abstract Meaning Representation-Based Logic-Driven Data Augmentation for Logical Reasoning
Strong AI Lab, NAOInstitute, Waipapa Taumata Rau - The University of Auckland(强人工智能实验室、NAO研究所、怀帕帕·陶马陶·劳 - 奥克兰大学);Xtracta, New Zealand(新西兰Xtracta公司);University of Cambridge, UK(英国剑桥大学);Sun Yat-Sen University, China(中国中山大学);The University of Auckland, New Zealand(新西兰奥克兰大学)
Deep Retrieval at CheckThat! 2025: Identifying Scientific Papers from Implicit Social Media Mentions via Hybrid Retrieval and Re-Ranking
University of Zurich(苏黎世大学);Centre for Artificial Intelligence(人工智能中心);Zurich University of Applied Sciences(应用科学大学);Institute of Neuroinformatics(神经信息学研究所);European Centre for Living Technology(欧洲生活技术中心)
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
Microsoft Research(微软研究院);Microsoft Dynamics 365 AI(微软 Dynamics 365 AI)
本文探讨了使用知识蒸馏来改进多任务深度神经网络(MT-DNN)(Liu et al.,2019),用于跨多个自然语言理解任务学习文本表示。虽然集成学习可以提高模型性能,但服务于大型DNN(如MT-DNN)的集成可能非常昂贵。在这里,我们应用知识蒸馏方法(Hinton等人,2015年,在多任务学习环境中。对于每个任务,我们训练不同MT-DNN(教师)的集合,其性能优于任何单个模型,然后通过多任务学习来训练单个MT-DNN(学生)以从这些集合教师获得\emph{distill}知识。我们发现,经过提炼的MT-DNN在9个GLUE任务中的7个上显著优于原始MT-DNN,将GLUE基准测试(单个模型)推高到83.7%(1.5%的绝对改进\footnote{ Based on the GLUE leaderboard at https://gluebenchmark.com/leaderboard as of April 1, 2019.})。代码和预训练模型将在https://github.com/namisan/mt-dnn上公开提供。
RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark
Sberbank(俄罗斯联邦储蓄银行);National Research University Higher School of Economics(国立研究型大学高等经济学院);Moscow Institute of Physics and Technology(莫斯科物理技术学院);Huawei Noah’s Ark lab(华为诺亚方舟实验室)
Evaluating and Improving the Coreference Capabilities of Machine Translation Models
The Hebrew University of Jerusalem(希伯来大学);Bar Ilan University(巴伊兰大学)
机器翻译(MT)需要广泛的语言能力,当前的端到端模型期望通过观察双语语料库中对齐的句子来隐式地学习这些语言能力。在这项工作中,我们要求:\emph{How well do MT models learn coreference resolution from implicit signal?}为了回答这个问题,我们开发了一种评估方法,从MT输出中获得共指聚类,并在不需要目标语言注释的情况下对其进行评估。我们进一步评估了几个著名的开源和商业MT系统,从英语翻译到六种目标语言,并将它们与三个具有挑战性的基准测试中最先进的共指消解器进行比较。我们的研究结果表明,单语解析器大大优于MT模型。受此结果的启发,我们实验了不同的方法,将共指消解模型的输出纳入MT中,显示出对强基线的改进。