BLADE: Benchmarking Language Model Agents for Data-Driven Science
University of Washington(华盛顿大学);UC Berkeley(伯克利大学);New York University(纽约大学);Stanford University(斯坦福大学);University of British Columbia(不列颠哥伦比亚大学);Microsoft(微软);George Washington University(乔治·华盛顿大学)
Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu… 展开作者
Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu, Mike A. Merrill, Jeffrey Heer, Tim Althoff
Data-driven scientific discovery requires the iterative integration of scientific domain knowledge, statistical expertise, and an understanding of data semantics to make nuanced analytical decisions, e.g., about which variables, transformations, and statistical models to consider. LM-based agents equipped with planning, memory, and code execution capabilities have the potential to support data-driven science. However, evaluating agents on such open-ended tasks is challenging due to multiple valid approaches, partially correct steps, and different ways to express the same decisions. To address these challenges, we present BLADE, a benchmark to automatically evaluate agents' multifaceted approaches to open-ended research questions. BLADE consists of 12 datasets and research questions drawn from existing scientific literature, with ground truth collected from independent analyses by expert data scientists and researchers. To automatically evaluate agent responses, we developed corresponding computational methods to match different representations of analyses to this ground truth. Though language models possess considerable world knowledge, our evaluation shows that they are often limited to basic analyses. However, agents capable of interacting with the underlying data demonstrate improved, but still non-optimal, diversity in their analytical decision making. Our work enables the evaluation of agents for data-driven science and provides researchers deeper insights into agents' analysis approaches.
AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite
Asta Team, Allen Institute for AI(Asta团队,人工智能研究所);University of Maryland(马里兰大学);University of Arizona(亚利桑那大学);University of Zurich(苏黎世大学);Bar-Ilan University(巴伊兰大学);University of Washington(华盛顿大学)
Into the Unknown: Benchmarking AI agents on open scientific research problems
Carnegie Mellon University(卡内基梅隆大学);University of Minnesota(明尼苏达大学);Stanford University(斯坦福大学);Trillion Labs;Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室);Sandia National Laboratory(桑迪亚国家实验室);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);Ames National Laboratory(艾姆斯国家实验室);Iowa State University(爱荷华州立大学);University of Colorado Boulder(科罗拉多大学博尔德分校);Michigan State University(密歇根州立大学);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室);TelePIX;Johns Hopkins University(约翰斯·霍普金斯大学);NEC Laboratories Europe(NEC欧洲实验室);Ss.Cyril and Methodius University of Skopje(斯科普里圣西里尔和圣美多德大学)
本研究推出Into the Unknown基准,测试9个前沿智能体撰写开放科学问题后续论文的能力,发现无智能体达到领域科学家标准,仍有较大提升空间。
TCS-BENCH: Benchmarking State-of-the-Art Generative AI Theoretical Computer Science Research Ability
Google Research(谷歌研究院);CNRS(法国国家科学研究中心);IRIF(法国信息学研究所);Université Paris-Cité(巴黎城市大学);Princeton University(普林斯顿大学);Université Paris-Saclay(巴黎萨克雷大学);CEA(法国原子能和替代能源委员会);California Institute of Technology(加州理工学院);Google DeepMind(谷歌DeepMind)
MatTools: Benchmarking Large Language Models for Materials Science Tools
Center for Structural Materials, Department of Mechanical Engineering, The University of Hong Kong(香港大学机械工程系结构材料中心);Materials Innovation Institute for Life Sciences(生命科学创新研究所)
ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences
University of Notre Dame(圣母大学);Old Dominion University(老道明大学);Pennsylvania State University(宾夕法尼亚州立大学);Independent Researcher(独立研究员);Uppsala University(乌普萨拉大学);Center for Open Science(开放科学中心)
SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Models
Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学);Hefei University of Technology(合肥工业大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州))
HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models
National Engineering Research Center for Software Engineering, Peking University(北京大学软件工程国家工程研究中心);Institute of Artificial Intelligence, China Telecom (TeleAI)(中国电信人工智能研究院);Tsinghua University(清华大学);Chinese Academy of Sciences(中国科学院);University of British Columbia(不列颠哥伦比亚大学);Renmin University of China(中国人民大学)
SciHorizon-GENE: Benchmarking LLM for Life Sciences Inference from Gene Knowledge to Functional Understanding
Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);University of the Chinese Academy of Sciences(中国科学院大学);DUKE-NUS Medical School, National University of Singapore(新加坡国立大学杜克-新加坡医学学校);Singapore Immunology Network, Agency for Science, Technology and Research(新加坡免疫网络,科技研究局)
LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Prediction
Princeton University(普林斯顿大学);Vertaix;Acceleration Consortium, University of Toronto(多伦多大学加速联盟);Vector Institute for Artificial Intelligence(向量人工智能研究所);Schwartz Reisman Institute for Technology and Society(施瓦茨·赖斯曼技术与社会研究所);University of Toronto(多伦多大学)
Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction
University of Chinese Academy of Sciences(中国科学院大学);Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室)
HyperGVL: Benchmarking and Improving Large Vision-Language Models in Hypergraph Understanding and Reasoning
Southern University of Science and Technology(南方科技大学);Hong Kong University of Science and Technology(香港科技大学);Huawei Research(华为研究);Beihang University(北航)
PVFH-Bench: Benchmarking Premise-Induced Visual Fixation Hallucinations in Vision-Language Models
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);ModelBest Inc.(ModelBest公司);State Key Lab. of Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Institute for AI, Tsinghua University(清华大学人工智能研究院)