RWTH Aachen University（亚琛RWTH大学）；University of Oxford（牛津大学）；Mila – Quebec AI Institute（魁北克AI研究所）；Technion - Israel Institute of Technology（技术学院-以色列理工学院）；ETAS Research University of Stuttgart（斯图加特大学ETAS研究所）；Google Research（谷歌研究）；Microsoft Research（微软研究院）
Timo Stoll, Chendi Qian, Ben Finkelshtein, Ali Parviz, Darius Weber, Fabrizio Frasca, Hadar Shavit, Antoine Siraudin, Arman Mielke, Marie Anastacio, Erik Müller… 展开作者
Timo Stoll, Chendi Qian, Ben Finkelshtein, Ali Parviz, Darius Weber, Fabrizio Frasca, Hadar Shavit, Antoine Siraudin, Arman Mielke, Marie Anastacio, Erik Müller, Maya Bechler-Speicher, Michael Bronstein, Mikhail Galkin, Holger Hoos, Mathias Niepert, Bryan Perozzi, Jan Tönshoff, Christopher Morris
Machine learning on graphs has made substantial progress across domains such as molecular property prediction and chip design. Yet benchmarking practices remain fragmented, often relying on narrow, task-specific datasets and inconsistent evaluation protocols, hindering reproducibility and broader progress. With the recent popularity of graph foundation models, these weaknesses have become apparent, as existing benchmarks are insufficient for thorough evaluation. To address these challenges, we introduce GraphBench, a comprehensive benchmark suite spanning diverse real-world domains and task settings, including node-level, edge-level, graph-level, and generative tasks. GraphBench provides standardized evaluation protocols, including consistent dataset splits and metrics for assessing out-of-distribution generalization across selected tasks, as well as a unified hyperparameter-tuning framework. We further evaluate GraphBench with recent message-passing neural networks and graph transformer models, establishing principled baselines for future research. See www.graphbench.io for further details.
We introduce Dynaboard, an evaluation-as-a-service framework for hosting benchmarks and conducting holistic model comparison, integrated with the Dynabench platform. Our platform evaluates NLP models directly instead of relying on self-reported metrics or predictions on a single dataset. Under this paradigm, models are submitted to be evaluated in the cloud, circumventing the issues of reproducibility, accessibility, and backwards compatibility that often hinder benchmarking in NLP. This allows users to interact with uploaded models in real time to assess their quality, and permits the collect…
Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approaches
Applied AI（应用人工智能）；Artificial Intelligence Research Institute（人工智能研究所）；HSE University（俄罗斯高等经济大学）；Sb AI Lab（Sb人工智能实验室）；MWS AI（MWS人工智能）；Yandex；Independent Researcher（独立研究者）
Astex Pharmaceuticals（Astex制药公司）；University of Cambridge（剑桥大学）；The Institute of Science and Technology Austria（奥地利科学技术研究所）
We introduce a machine-learning (ML) framework for high-throughput benchmarking of diverse representations of chemical systems against datasets of materials and molecules. The guiding principle underlying the benchmarking approach is to evaluate raw descriptor performance by limiting model complexity to simple regression schemes while enforcing best ML practices, allowing for unbiased hyperparameter optimization, and assessing learning progress through learning curves along series of synchronized train-test splits. The resulting models are intended as baselines that can inform future method de…
GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments
Yale University（耶鲁大学）；Zhejiang University（浙江大学）；China University of Geosciences（中国地质大学）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Tongji University（同济大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
RealBench: Benchmarking Data-Driven Numerical Weather Forecasting Under Operational Conditions and Extreme Event Challenges
The Hong Kong University of Science and Technology（香港科学与技术大学）；Nanjing University（南京大学）；Southern University of Science and Technology（南方科技大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai TechWind Technology Co., Ltd.（上海技风科技有限公司）
Nanyang Technological University（南洋理工大学）；NVIDIA AI Tech Center（英伟达人工智能技术中心）；Nokia Bell Lab.（诺基亚贝尔实验室）
Deep learning (DL) models have become core modules for many applications. However, deploying these models without careful performance benchmarking that considers both hardware and software's impact often leads to poor service and costly operational expenditure. To facilitate DL models' deployment, we implement an automatic and comprehensive benchmark system for DL developers. To accomplish benchmark-related tasks, the developers only need to prepare a configuration file consisting of a few lines of code. Our system, deployed to a leader server in DL clusters, will dispatch users' benchmark job…
New Benchmarking Shows Limited Generalization Power of TCR Antigenic Epitope Prediction Models
Trustworthy and Intelligent Computing Lab (TAIC), Department of Computer Science and Electrical Engineering, University of Maryland, Baltimore County（可信智能计算实验室（TAIC），计算机科学与电气工程系，马里兰大学巴尔的摩分校）；Children’s Hospital of Philadelphia（费城儿童医院）；Department of Bioengineering, University of Pennsylvania（生物工程系，宾夕法尼亚大学）；Institute for Immunology & Immune Health, University of Pennsylvania（免疫学与免疫健康研究所，宾夕法尼亚大学）；Institute for RNA Innovation, University of Pennsylvania（RNA创新研究所，宾夕法尼亚大学）；Abramson Cancer Center, University of Pennsylvania（Abramson癌症中心，宾夕法尼亚大学）；Center for Precision Engineering for Health, University of Pennsylvania（健康精准工程中心，宾夕法尼亚大学）；Center for Cellular Immunotherapies, University of Pennsylvania（细胞免疫治疗中心，宾夕法尼亚大学）
Benchmarking World-Model Learning with Environment-Level Queries
Basis Research Institute；DFKI GmbH；Harvard University；Universit\'e de Montr\'eal \& Mila - Quebec AI Institute；University of Cambridge；Massachusetts Institute of Technology；Cornell University
Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility
Shanghai Jiao Tong University（上海交通大学）；OpenDataLab, Shanghai Artificial Intelligence Laboratory（OpenDataLab，上海人工智能实验室）；The University of Hong Kong（香港大学）；Peking University（北京大学）
LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering
DreamX Team, Alibaba Group（达摩院团队，阿里巴巴集团）；Beijing University of Posts and Telecommunications（北京邮电大学）；UNSW Sydney（新南威尔士大学悉尼分校）；Data61, CSIRO（联邦科学与工业研究组织数据61研究所）
HealthSLM-Bench: Benchmarking Small Language Models for Mobile and Wearable Healthcare Monitoring
School of Computing and Information Systems, University of Melbourne, Australia（墨尔本大学计算机与信息系统学院）；School of Computer Science, University of Auckland, New Zealand（奥克兰大学计算机科学学院）
Agents Don't Just Agree, They Remember: Benchmarking Persistent Sycophancy in Self-Improving Personal Agents
City University of Hong Kong（香港城市大学）；ByteDance（字节跳动）；Yale University（耶鲁大学）；Fudan University（复旦大学）；Hong Kong Baptist University（香港浸会大学）；Dalian University of Technology（大连理工大学）
OptumLabs at UnitedHealth Group（联合健康集团 OptumLabs）；University of Pennsylvania（宾夕法尼亚大学）
We ascertain and compare the performances of AutoML tools on large, highly imbalanced healthcare datasets. We generated a large dataset using historical administrative claims including demographic information and flags for disease codes in four different time windows prior to 2019. We then trained three AutoML tools on this dataset to predict six different disease outcomes in 2019 and evaluated model performances on several metrics. The AutoML tools showed improvement from the baseline random forest model but did not differ significantly from each other. All models recorded low area under the …
OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scaling
Shanghai University of Finance and Economics（上海财经大学）；Booth School of Business, University of Chicago（芝加哥大学商学院）；Antai School of Economics and Management, Shanghai Jiao Tong University（上海交通大学安泰经济管理学院）
Pioneering data profiling systems such as Metanome and OpenClean brought public attention to science-intensive data profiling. This type of profiling aims to extract complex patterns (primitives) such as functional dependencies, data constraints, association rules, and others. However, these tools are research prototypes rather than production-ready systems. The following work presents Desbordante - a high-performance science-intensive data profiler with open source code. Unlike similar systems, it is built with emphasis on industrial application in a multi-user environment. It is efficient, r…
Benchmarking Pathology Foundation Models for Spatial Domain Understanding
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Brainnetome Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所脑网膜工程中心）；Beijing Key Laboratory of Brainnetome and Brain-Computer Interface, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所北京脑网膜与脑机接口重点实验室）；DAMO Academy, Alibaba Group（阿里云达摩院）；ShanghaiTech University（上海科技大学）
OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences
University of Science and Technology of China（中国科学技术大学）；Institute of Theoretical Physics, Chinese Academy of Sciences（中国科学院理论物理研究所）；Hefei National Laboratory（合肥国家实验室）；Hefei National Research Center for Physical Sciences at the Microscale（合肥微尺度物质科学国家研究中心）；School of Physical Sciences, University of Science and Technology of China（中国科学技术大学物理学院）；Institute for Advanced Algorithms Research（先进算法研究院）；Endless Frontier（无尽前沿（Endless Frontier））
Forecasting MBTA Transit Dynamics: A Performance Benchmarking of Statistical and Machine Learning Models
Sunset High School（Sunset 高中）；Woodbridge High School（Woodbridge 高中）；Western Academy of Beijing（北京西方学院）；Computer Science Department, Metropolitan College（Metropolitan College 计算机科学系）
AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science
School of Statistics, University of Minnesota（明尼苏达大学统计学系）；AIScientists, Inc.（AIScientists公司）；Data Science Institute, University of Chicago（芝加哥大学数据科学研究所）；Carlson School of Management, University of Minnesota（明尼苏达大学卡尔森管理学院）；Cisco Research（思科研究）；Department of Electrical and Computer Engineering, University of Minnesota（明尼苏达大学电气与计算机工程系）；Division of Computational Health Sciences, University of Minnesota（明尼苏达大学计算健康科学 division）
Pix2Fact: When Vision Is Not Enough -- Benchmarking Fine-Grained VQA with Web Verification on High-Resolution Real-World Scenes
GADE Union (Global AI Data Experts Union)（GADE联盟（全球人工智能数据专家联盟））；Shanghai Jiao Tong University（上海交通大学）；Nanyang Technological University（南洋理工大学）；New York University（纽约大学）；Cambridge University（剑桥大学）；The University of Hong Kong（香港大学）
Benchmarking neural surrogates on realistic spatiotemporal multiphysics flows
State Key Laboratory for Turbulence and Complex Systems, School of Mechanics and Engineering Science, Peking University（湍流与复杂系统国家重点实验室，力学与工程科学学院，北京大学）；Gaoling School of Artificial Intelligence, Renmin University of China（Gallagher人工智能学院，中国人民大学）；AI for Science Institute（人工智能科学研究院）；University of Calgary（卡尔加里大学）；Kyoto University（京都大学）；FM Global（FM全球）；LandSpace Technology Corporation Ltd.（陆地方向技术有限公司）；Aero Engine Academy of China（中国航空发动机学院）
MedImageOSWorld: Benchmarking GUI Agents for Medical Image Consoles
Biomedical Imaging Research Institute, Cedars-Sinai Medical Center（西达赛奈医疗中心生物医学成像研究所）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Technical University of Munich（慕尼黑工业大学）；Berlin Ultrahigh Field Facility, Max Delbrück Center for Molecular Medicine in the Helmholtz Association（亥姆霍兹协会马克斯·德尔布吕克分子医学中心柏林超高场设施）
Sionna is a GPU-accelerated open-source library for link-level simulations based on TensorFlow. It enables the rapid prototyping of complex communication system architectures and provides native support for the integration of neural networks. Sionna implements a wide breadth of carefully tested state-of-the-art algorithms that can be used for benchmarking and end-to-end performance evaluation. This allows researchers to focus on their research, making it more impactful and reproducible, while saving time implementing components outside their area of expertise. This white paper provides a brief…
