AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge
Nanyang Technological University（南洋理工大学）；University of California, Santa Barbara（加州大学圣芭芭拉分校）；National University of Singapore（新加坡国立大学）；Shanghai Jiao Tong University（上海交通大学）；University of Arizona（亚利桑那大学）
Xiaobao Wu, Liangming Pan, Yuxi Xie, Ruiwen Zhou, Shuai Zhao, Yubo Ma, Mingzhe Du, Rui Mao, Anh Tuan Luu, William Yang Wang
Data contamination hinders fair LLM evaluation by introducing test data into newer models' training sets. Existing studies solve this challenge by updating benchmarks with newly collected data. However, they fail to guarantee contamination-free evaluation as the newly collected data may contain pre-existing knowledge, and their benchmark updates rely on intensive human labor. To address these issues, we in this paper propose AntiLeak-Bench, an automated anti-leakage benchmarking framework. Instead of simply using newly collected data, we construct samples with explicitly new knowledge absent from LLMs' training sets, which thus ensures strictly contamination-free evaluation. We further design a fully automated workflow to build and update our benchmark without human labor. This significantly reduces the cost of benchmark maintenance to accommodate emerging LLMs. Through extensive experiments, we highlight that data contamination likely exists before LLMs' cutoff time and demonstrate AntiLeak-Bench effectively overcomes this challenge.
CoreEval: Automatically Building Contamination-Resilient Datasets with Real-World Knowledge toward Reliable LLM Evaluation
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）；Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies（广东省新型安全智能技术重点实验室）；The Chinese University of Hong Kong（香港中文大学）；The Hong Kong Polytechnic University（香港理工大学）
Luddy School of Informatics, Computing & Engineering（卢迪信息学、计算与工程学院）；Center for Complex Networks & Systems（复杂网络与系统中心）；Indiana University Network Science Institute（印第安纳大学网络科学研究所）
LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?
New York University（纽约大学）；Princeton University（普林斯顿大学）；University of California San Diego（加利福尼亚大学圣地亚哥分校）；University of Washington（华盛顿大学）；Canyon Crest Academy（峡谷克雷斯特学院）；University of Waterloo（滑铁卢大学）；McGill University（麦吉尔大学）
Energy-Arena: A Dynamic Benchmark for Operational Energy Forecasting
Karlsruhe Institute of Technology, Institute for Industrial Production（卡尔斯鲁厄理工学院，工业生产研究所）；University of Duisburg-Essen, House of Energy Markets and Finance（杜伊斯堡-埃森大学，能源市场与金融中心）；Karlsruhe Institute of Technology, Institute for Automation and Applied Informatics（卡尔斯鲁厄理工学院，自动化与应用信息研究所）；Karlsruhe Institute of Technology, Institute of Statistics（卡尔斯鲁厄理工学院，统计研究所）；Heidelberg Institute for Theoretical Studies（海德堡理论研究所）；Marburg University, Department of Mathematics and Computer Science（马尔堡大学，数学与计算机科学系）；Volatile GmbH（Volatile GmbH公司）；Forschungszentrum Jülich GmbH, Institute of Climate and Energy Systems – Jülich Systems Analysis（焦耳研究中心 GmbH，气候与能源系统研究所——焦耳系统分析）
