The versatility of large language models (LLMs) led to the creation of diverse benchmarks that thoroughly test a variety of language models' abilities. These benchmarks consist of tens of thousands of examples making evaluation of LLMs very expensive. In this paper, we investigate strategies to reduce the number of evaluations needed to assess the performance of an LLM on several key benchmarks. For example, we show that to accurately estimate the performance of an LLM on MMLU, a popular multiple-choice QA benchmark consisting of 14K examples, it is sufficient to evaluate this LLM on 100 curated examples. We release evaluation tools and tiny versions of popular benchmarks: Open LLM Leaderboard, MMLU, HELM, and AlpacaEval 2.0. Our empirical analysis demonstrates that these tools and tiny benchmarks are sufficient to reliably and efficiently reproduce the original evaluation results.
Can Language Models Analyze Data? Evaluating Large Language Models for Question Answering over Datasets
School of Production Engineering and Management, Technical University of Crete（生产工程与管理学院，希腊克里特技术大学）；Institute of Computer Science, Foundation for Research and Technology - Hellas（计算机科学研究所，希腊基础研究与技术研究院）
FarsEval-PKBETS: A new diverse benchmark for evaluating Persian large language models
Shahid Beheshti University（沙希德·贝赫什提大学）；Amirkabir University of Technology（阿米尔卡比尔理工大学）；Qom University（库姆大学）；Islamic Azad University（伊斯兰阿扎德大学）；Sharif University of Technology（谢里夫理工大学）
Let's Have a Conversation: Designing and Evaluating LLM Agents for Interactive Optimization
Operations Research Center and Sloan School of Management, Massachusetts Institute of Technology（麻省理工学院运筹学研究中心和斯隆管理学院）；Kellogg School of Management, Northwestern University（西北大学凯洛格管理学院）
Evaluating Reliability Asymmetries in Chinese Factual Search and AI Answers
Department of Electronics, Information and Bioengineering, Politecnico di Milano（电子、信息与生物工程系，米兰理工学院）；University of Science and Technology of China（中国科学技术大学）
T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models
UCAS（中国科学院大学）；Tsinghua University（清华大学）；Institute for AI（人工智能研究院）；Tsinghua-Bosch Joint ML Center（清华-博世联合机器学习中心）；THBI Lab（THBI实验室）；BNRist Center（BNRist中心）；RealAI
Who Owns That? Evaluating Ownership Intuitions in Large Language Models
College of AI, Tsinghua University（清华大学人工智能学院）；Department of Psychological and Cognitive Sciences, Tsinghua University（清华大学心理与认知科学系）；Faculty of Psychology, Beijing Normal University（北京师范大学心理学部）
Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Loyola University Chicago（芝加哥洛约拉大学）；Boston Children’s Hospital Harvard Medical School（波士顿儿童医院哈佛医学院）；University of Colorado-Anschutz（科罗拉多大学安舒茨分校）
Iterative Learning of Computable Phenotypes for Treatment Resistant Hypertension using Large Language Models
Federal University of ABC（亚巴西联邦大学）；Department of Pathology and Laboratory Medicine, University of Pennsylvania（宾夕法尼亚大学病理学与实验室医学系）；Computational Health Informatics Program, Boston Children’s Hospital, Harvard Medical School（哈佛医学院波士顿儿童医院计算健康信息学项目）
FactEHR: A Dataset for Evaluating Factuality in Clinical Notes Using LLMs
Khoury College of Computer Sciences, Northeastern University（东北大学克劳尔计算机科学学院）；Center for Biomedical Informatics Research, Stanford University（斯坦福大学生物医学信息学研究中心）；Department of Biomedical Data Science, Stanford School of Medicine（斯坦福医学院生物医学数据科学系）；Stanford Health Care（斯坦福健康系统）；Department of Medicine, Stanford School of Medicine（斯坦福医学院医学系）；Clinical Excellence Research Center, Stanford School of Medicine（斯坦福医学院临床卓越研究中心）；Department of Anesthesiology, Perioperative & Pain Medicine, Stanford School of Medicine（斯坦福医学院麻醉学、手术及疼痛医学系）；Department of Dermatology, Stanford School of Medicine（斯坦福医学院皮肤病学系）；Technology and Digital Solutions, Stanford Health Care（斯坦福健康系统技术与数字解决方案）
Evaluating Large Language Models for Automated Clinical Abstraction in Pulmonary Embolism Registries: Performance Across Model Sizes, Versions, and Parameters
Duke University Health System（杜克大学医疗系统）；Duke University（杜克大学）；Duke Clinical Research Institute（杜克临床研究所）
Evaluating Contextualized Language Models for Hungarian
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；Institute for Computer Science and Control（计算机科学与控制研究所）；Alfréd Rényi Institute of Mathematics（阿尔弗雷德·莱尼数学研究所）
