Large Language Model (LLM) leaderboards based on benchmark rankings are regularly used to guide practitioners in model selection. Often, the published leaderboard rankings are taken at face value - we show this is a (potentially costly) mistake. Under existing leaderboards, the relative performance of LLMs is highly sensitive to (often minute) details. We show that for popular multiple-choice question benchmarks (e.g., MMLU), minor perturbations to the benchmark, such as changing the order of choices or the method of answer selection, result in changes in rankings up to 8 positions. We explain this phenomenon by conducting systematic experiments over three broad categories of benchmark perturbations and identifying the sources of this behavior. Our analysis results in several best-practice recommendations, including the advantage of a hybrid scoring method for answer selection. Our study highlights the dangers of relying on simple benchmark evaluations and charts the path for more robust evaluation schemes on the existing benchmarks. The code for this paper is available at https://github.com/National-Center-for-AI-Saudi-Arabia/lm-evaluation-harness.
Latent Performance Profiling of Large Language Models
Department of Electrical Engineering, Indian Institute of Technology Delhi（印度理工学院德里分校电子工程系）；Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi（印度理工学院德里分校人工智能学院）；Hewlett Packard Enterprise, India（印度惠普企业公司）；Department of Computer Science & Engineering, Indian Institute of Technology Kharagpur（印度理工学院Khargapur分校计算机科学与工程系）；A.K.Choudhury School of Information Technology, University of Calcutta, India（印度加尔各答大学信息科技学院）；Department of Computer Science & Engineering, Indian Institute of Technology Bombay（印度理工学院孟买分校计算机科学与工程系）；Department of Computer Science, Ashoka University, India（阿什oka大学计算机科学系）；Department of Computer Science & Engineering, Indian Institute of Technology Jodhpur（印度理工学院朱罗普分校计算机科学与工程系）
GI-Bench: A Panoramic Benchmark Revealing the Knowledge-Experience Dissociation of Multimodal Large Language Models in Gastrointestinal Endoscopy Against Clinical Standards
Zhongshan Hospital, Fudan University（复旦大学附属中山医院）；School of Basic Medical Sciences, Fudan University（复旦大学基础医学院）；Microsoft Research Asia（微软亚洲研究院）；Data Science Institute, Imperial College London（帝国理工学院数据科学研究所）
LLMsPark: A Benchmark for Evaluating Large Language Models in Strategic Gaming Contexts
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；Tongji University（同济大学）；AIR, Tsinghua University（清华大学人工智能研究院）；BAAI（百度人工智能研究院）
Rethinking Domain Specialization for Open-Ended Scientific Reasoning in Astronomy Language Models
Oak Ridge National Laboratory（橡树岭国家实验室）；The Ohio State University（俄亥俄州立大学）；QUP/IPNS, High Energy Accelerator Research Organization (KEK)（高能加速器研究机构（KEK）QUP/IPNS）；Max-Planck-Institut für Astronomie（马克斯·普朗克天文研究所）
Doxing via the Lens: Revealing Location-related Privacy Leakage on Multi-modal Large Reasoning Models
University of Georgia（佐治亚大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；John Hopkins University（约翰·霍普金斯大学）；University of Southern California（南加州大学）；University of Maryland, College Park（马里兰大学学院市分校）
A Linguistically Motivated Analysis of Intonational Phrasing in Text-to-Speech Systems: Revealing Gaps in Syntactic Sensitivity
Institute for Logic, Language and Computation, University of Amsterdam（逻辑、语言与计算研究所，阿姆斯特丹大学）；Cognitive Science and Artificial Intelligence, Tilburg University（认知科学与人工智能，蒂尔堡大学）
Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks
MIT（麻省理工学院）；Harvard（哈佛大学）；Mass General Brigham（麻省总医院布里格姆医疗系统）；Universitat Pompeu Fabra（庞培法布拉大学）；Aarhus University（奥胡斯大学）；Maastricht University（马斯特里赫特大学）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；University of Virginia（弗吉尼亚大学）
