SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University（浙江大学Hangzhou全球科技创新中心）；ZJU-UIUC Institute, Zhejiang University（浙江大学UIUC研究院）
Large language models (LLMs) are playing an increasingly important role in scientific research, yet there remains a lack of comprehensive benchmarks to evaluate the breadth and depth of scientific knowledge embedded in these models. To address this gap, we introduce SciKnowEval, a large-scale dataset designed to systematically assess LLMs across five progressive levels of scientific understanding: memory, comprehension, reasoning, discernment, and application. SciKnowEval comprises 28K multi-level questions and solutions spanning biology, chemistry, physics, and materials science. Using this benchmark, we evaluate 20 leading open-source and proprietary LLMs. The results show that while proprietary models often achieve state-of-the-art performance, substantial challenges remain -- particularly in scientific reasoning and real-world application. We envision SciKnowEval as a standard benchmark for evaluating scientific capabilities in LLMs and as a catalyst for advancing more capable and reliable scientific language models.
SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models
ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University（浙江大学杭州国际科创中心）；The Polytechnic Institute, Zhejiang University（浙江大学理工学院）；ZJU-UIUC, Zhejiang University（浙江大学伊利诺伊大学厄巴纳-香槟联合学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；AntGroup（蚂蚁集团）
Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Bank of China（中国银行）；School of Engineering Science, University of Chinese Academy of Sciences（中国科学院大学工程科学学院）；State Key Laboratory of Nonlinear Mechanics, Institute of Mechanics, Chinese Academy of Sciences（中国科学院力学研究所非线性力学重点实验室）
EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMs
Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Hong Kong Polytechnic University（香港理工大学）；The Chinese University of Hong Kong（香港中文大学）
LSR-Ben: A Logical and Scientific Reasoning Benchmark for Evaluating Process Reward Models
Research Center for Social Computing（社会计算研究中心）；Interactive Robotics, Harbin Institute of Technology, China（交互机器人学，哈尔滨工业大学，中国）；Beijing Academy of Artificial Intelligence, Beijing, China（北京人工智能研究院，北京，中国）
CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning
Google（谷歌）；Harvard（哈佛大学）；University of Zurich（苏黎世大学）；NIST（国家标准与技术研究院）；UMD College Park（大学公园大学分校）；Rutgers（罗格斯大学）；FU Berlin（柏林自由大学）；Modelyst；Cornell（康奈尔大学）
ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows
The University of Hong Kong（香港大学）；Shanghai AI Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；Peking University（北京大学）；Nanjing University（南京大学）；East China Normal University（华东师范大学）；Yale University（耶鲁大学）
Hallucination as a Feature, not a Defect: Evaluating a multi-agent architecture to transform speculative language-model outputs into testable scientific hypotheses
AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials
University of New South Wales, NSW, Sydney, Australia（新南威尔士大学，新州，悉尼，澳大利亚）；Suzhou Institute for Advanced Research, University of Science（苏州先进研究院，科学大学）；Shanghai Artificial Intelligence Laboratory, Shanghai, China（上海人工智能实验室，上海，中国）；Cornell University（康奈尔大学）
提出AtomWorld基准，通过十种基本原子结构操作评估LLM在材料科学中的空间推理能力，发现Claude Opus 4.6表现最佳但复杂空间关系操作成功率低，表明LLM更适合作为辅助工具而非完全自主的科研代理。
Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences
Johns Hopkins University（约翰霍普金斯大学）；Columbia University（哥伦比亚大学）；The Hong Kong Polytechnic University（香港理工大学）；The Pennsylvania State University（宾夕法尼亚州立大学）
University of Illinois at Urbana Champaign（伊利诺伊大学香槟分校）；TIB Leibniz Information Centre for Science and Technology（莱布尼茨科学与技术信息中心）；L3S Research Center at Leibniz University of Hannover（汉诺威莱布尼茨大学L3S研究中心）
Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource African Languages
Imperial College London（伦敦帝国学院）；University of South Africa（南非大学）；University of Johannesburg（约翰内斯堡大学）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；Canada CIFAR AI Chair（加拿大CIFAR人工智能主席项目）；Masakhane General Purpose（Masakhane通用项目）；Saarland University（萨尔大学）；Equiano Institute（埃夸诺研究所）；OpenAI（OpenAI公司）
FlyAOC: Evaluating Agentic Ontology Curation of Drosophila Scientific Knowledge Bases
University of Michigan（密歇根大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, Berkeley（加州大学伯克利分校）；University of Washington（华盛顿大学）
Evaluating BERT-based Scientific Relation Classifiers for Scholarly Knowledge Graph Construction on Digital Library Collections
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；TIB Leibniz Information Centre for Science and Technology（莱布尼茨科学技术信息中心（TIB））；L3S Research Center at Leibniz University of Hannover（汉诺威莱布尼茨大学L3S研究中心）
Evaluating Compliance with Visualization Guidelines in Diagrams for Scientific Publications Using Large Vision Language Models
Department of Computer Science, University of Applied Sciences and Arts Dortmund（应用科学与艺术大学多特蒙德计算机科学系）；Institute for Medical Informatics, Biometry and Epidemiology (IMIBE)（医学信息学、生物统计与流行病学研究所）；Institute for Artificial Intelligence in Medicine (IKIM), University Hospital Essen（医学人工智能研究所，埃森大学医院）
