Geometric deep learning (GDL) has gained significant attention in scientific fields, for its proficiency in modeling data with intricate geometric structures. However, very few works have delved into its capability of tackling the distribution shift problem, a prevalent challenge in many applications. To bridge this gap, we propose GeSS, a comprehensive benchmark designed for evaluating the performance of GDL models in scientific scenarios with distribution shifts. Our evaluation datasets cover diverse scientific domains from particle physics, materials science to biochemistry, and encapsulate a broad spectrum of distribution shifts including conditional, covariate, and concept shifts. Furthermore, we study three levels of information access from the out-of-distribution (OOD) test data, including no OOD information, only unlabeled OOD data, and OOD data with a few labels. Overall, our benchmark results in 30 different experiment settings, and evaluates 3 GDL backbones and 11 learning algorithms in each setting. A thorough analysis of the evaluation results is provided, poised to illuminate insights for GDL researchers and domain practitioners who are to use GDL in their applications.
Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility
Shanghai Jiao Tong University(上海交通大学);OpenDataLab, Shanghai Artificial Intelligence Laboratory(OpenDataLab,上海人工智能实验室);The University of Hong Kong(香港大学);Peking University(北京大学)
Benchmarking Autonomy in Scientific Experiments: A Hierarchical Taxonomy for Autonomous Large-Scale Facilities
University of Greenwich, Old Royal Naval College, Park Row, London, SE10 9LS, United Kingdom(格林威治大学,皇家海军学院,伦敦公园行,SE10 9LS,英国);ISIS Neutron & Muon Source, Rutherford Appleton Laboratory, Didcot, OX11 0QX, United Kingdom(ISIS中子与穆子源,拉瑟福德-苹果顿实验室,迪茨,OX11 0QX,英国);The Faraday Institution, Harwell Science and Innovation Campus, Didcot, OX11 0RA, United Kingdom(法拉第机构,哈威尔科学与创新校园,迪茨,OX11 0RA,英国)
LabInstruct: Benchmarking Situated Instructional Video Generation for Lab Procedures
Central South University(中南大学);National University of Singapore(新加坡国立大学);Beijing University of Posts and Telecommunications(北京邮电大学);Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences(中国科学院重庆绿色智能技术研究院);Monash University(莫纳什大学)
Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
Yale University(耶鲁大学);Broad Institute of MIT and Harvard(布罗德研究所);The Pennsylvania State University(宾夕法尼亚州立大学);Northeastern University(东北大学);Northwestern University(西北大学)
MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems
Department of Computer Science, Yale University(耶鲁大学计算机科学系);Department of Computer Science, Brown University(布朗大学计算机科学系);School of Data Science, Fudan University(复旦大学数据科学学院);Shanghai Key Laboratory of Data Science(上海数据科学关键实验室);Datawiz LLC
Rutherford Appleton Laboratory, Science and Technology Facilities Council(卢瑟福·阿普尔顿实验室,科学与技术设施委员会);Oak Ridge National Laboratory(橡树岭国家实验室);University of Virginia(弗吉尼亚大学)
Into the Unknown: Benchmarking AI agents on open scientific research problems
Carnegie Mellon University(卡内基梅隆大学);University of Minnesota(明尼苏达大学);Stanford University(斯坦福大学);Trillion Labs;Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室);Sandia National Laboratory(桑迪亚国家实验室);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);Ames National Laboratory(艾姆斯国家实验室);Iowa State University(爱荷华州立大学);University of Colorado Boulder(科罗拉多大学博尔德分校);Michigan State University(密歇根州立大学);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室);TelePIX;Johns Hopkins University(约翰斯·霍普金斯大学);NEC Laboratories Europe(NEC欧洲实验室);Ss.Cyril and Methodius University of Skopje(斯科普里圣西里尔和圣美多德大学)
本研究推出Into the Unknown基准,测试9个前沿智能体撰写开放科学问题后续论文的能力,发现无智能体达到领域科学家标准,仍有较大提升空间。
Multiphysics Bench: Benchmarking and Investigating Scientific Machine Learning for Multiphysics PDEs
xLeaF Lab, Hong Kong University of Science and Technology (Guangzhou)(xLeaF实验室,香港科技大学(广州));National University of Singapore(新加坡国立大学);AI for Science, Shanghai AI Lab(AI for Science,上海人工智能实验室)
SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Models
Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学);Hefei University of Technology(合肥工业大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州))
PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors
Center of Excellence for Generative AI, KAUST(生成人工智能卓越中心,KAUST);The Swiss AI Lab, IDSIA-USI/SUPSI(瑞士人工智能实验室,IDSIA-USI/SUPSI);NNAISENSE Principia-AI/PhysGym(NNAISENSE 原初人工智能/PhysGym)
CausalGame: Benchmarking Causal Thinking of LLM Agents in Games
MBZUAI(穆罕默德·本·扎耶德人工智能大学);Carnegie Mellon University(卡内基梅隆大学);Hong Kong Baptist University(香港浸会大学);University of Oxford(牛津大学);New York University, Abu Dhabi(纽约大学阿布扎比分校)
Does AI for science need another ImageNet Or totally different benchmarks? A case study of machine learning force fields
Institute of Computing Technology Chinese Academy of Science(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Microsoft Research(微软研究院)
本文以机器学习力场为案例,指出传统 AI 基准测试无法应对 AI4S 的分布外挑战,并提出涵盖样本效率、时域敏感性和跨数据集泛化的新基准方法,作为 SAIBench 项目的一部分。
DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking
Princeton University(普林斯顿大学);Boston University(波士顿大学);New York University(纽约大学);Flatiron Institute(Flatiron研究所);Institute for Advanced Studies(高级研究研究所)
InnoGym: Benchmarking the Innovation Potential of AI Agents
Zhejiang University(浙江大学);Ant Group(蚂蚁集团);National University of Singapore(新加坡国立大学);Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph(知识图谱联合实验室)
Benchmarking energy consumption and latency for neuromorphic computing in condensed matter and particle physics
University of Twente(特文特大学);Radboud University(拉德堡德大学);IBM Research Europe - Zürich(IBM欧洲研究院-苏黎世);Nikhef(荷兰国家亚原子物理研究所);Eidgenössische Technische Hochschule Zürich(苏黎世联邦理工学院);SURF Cooperation(SURF合作组织);University of Valencia-CSIC(瓦伦西亚大学-西班牙国家研究委员会)
Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction
New High Energy Theory Center(新高能理论中心);Department of Physics & Astronomy(物理与天文学系);Rutgers University(罗格斯大学);Faculty of Computing & Data Sciences(计算与数据科学学院)