Evaluation for many natural language understanding (NLU) tasks is broken: Unreliable and biased systems score so highly on standard benchmarks that there is little room for researchers who develop better systems to demonstrate their improvements. The recent trend to abandon IID benchmarks in favor of adversarially-constructed, out-of-distribution test sets ensures that current models will perform poorly, but ultimately only obscures the abilities that we want our benchmarks to measure. In this position paper, we lay out four criteria that we argue NLU benchmarks should meet. We argue most current benchmarks fail at these criteria, and that adversarial data collection does not meaningfully address the causes of these failures. Instead, restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, the reliability with which they are annotated, their size, and the ways they handle social bias.
Benchmarking Differential Privacy and Federated Learning for BERT Models
Manipal Institute of Technology（马尼帕尔理工学院）；Carnegie Mellon University（卡内基梅隆大学）；OpenMined；Yildiz Technical University（耶尔德兹技术大学）；Ford Motor Company（福特汽车公司）；University of California, San Diego（加州大学圣地亚哥分校）
Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets
University of Tokyo（东京大学）；University College London（伦敦大学学院）；Tohoku University（东北大学）；RIKEN Center for AIP（理化学研究所人工智能中心）；National Institute of Informatics（国立信息学研究所）
PVFH-Bench: Benchmarking Premise-Induced Visual Fixation Hallucinations in Vision-Language Models
School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；ModelBest Inc.（ModelBest公司）；State Key Lab. of Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Institute for AI, Tsinghua University（清华大学人工智能研究院）
Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents
Beijing University of Posts and Telecommunications（北京邮电大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Noumena AI
