Agentic AI offers a promising path to automating software development bottlenecks in scientific research pipelines, particularly for stages that take domain experts days to months to build and where correctness and robustness matter more than implementation details. We present an empirical study of general-purpose coding agents on a fly optogenetics data-to-discovery pipeline. We assess agents on tasks and datasets substantially larger than existing benchmarks and evaluation criteria grounded in domain expert standards. We show that agents can solve several pipeline stages, suggesting stage-level automation is tractable. By analyzing agents' code iterations, we show they struggle most without a pre-defined criterion, when they must instead use their scientific judgment to assess their current solution. Mirroring scientists, they sometimes attempt visual inspection of intermediate outputs for self-evaluation, but largely fail to interpret what they see or act on it appropriately. Solving the end-to-end pipeline requires stringing together successes across all stages, which is beyond agents' current abilities. We identify challenges largely absent from existing benchmarks, including computational resource management and generalization to held-out data. Finally, we distill principles for constructing scientific tasks and rigorous evaluation criteria for open-ended problems.
Mind Meets Space: Rethinking Agentic Spatial Intelligence from a Neuroscience-inspired Perspective
School of EEE, Nanyang Technological University(南洋理工大学电子工程系);Department of Civil Engineering, Tsinghua University(清华大学土木工程系);Dr. B. R. Ambedkar National Institute of Technology, Jalandhar(B.R.阿姆贝卡尔国家理工学院,贾兰德赫);KTH Royal Institute of Technology(皇家理工学院);College of AI, Tsinghua University(清华大学人工智能学院);CCDS, Nanyang Technological University(南洋理工大学CCDS)
An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery
Department of Statistics, Virginia Tech(统计学系,弗吉尼亚理工学院);Department of Statistical Science, Baylor University(统计科学系,贝勒大学);Advanced Research Computing, Virginia Tech(高级研究计算,弗吉尼亚理工学院)
SeekBrain: An Autonomous Multi-Agent System for Accelerating Neuroscience Discovery
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences(中国科学院脑科学与智能技术卓越创新中心);The Chinese University of Hong Kong(香港中文大学);Southeast University(东南大学);Fudan University(复旦大学);New York University(纽约大学)
ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows
The University of Hong Kong(香港大学);Shanghai AI Laboratory(上海人工智能实验室);Fudan University(复旦大学);Peking University(北京大学);Nanjing University(南京大学);East China Normal University(华东师范大学);Yale University(耶鲁大学)
Entity-Augmented Neuroscience Knowledge Retrieval Using Ontology and Semantic Understanding Capability of LLM
Sudha Gopalakrishnan Brain Centre, Indian Institute of Technology, Madras(印度理工学院马德拉斯分校苏达·戈帕拉克里希南脑科学中心);Department of Electrical Engineering, Indian Institute of Technology, Madras(印度理工学院马德拉斯分校电气工程系)
Evaluating Cross-Lingual Classification Approaches Enabling Topic Discovery for Multilingual Social Media Data
Centre for Data Science, School of Computer Science, Queensland University of Technology, Brisbane, QLD 4000, Australia(数据科学中心,计算机科学学院,昆士兰理工大学,布里斯班,QLD 4000,澳大利亚)
Budget-Sensitive Discovery Scoring: A Formally Verified Framework for Evaluating AI-Guided Scientific Selection
Indian Institute of Information Technology Allahabad (IIITA)(印度阿瓦德理工学院信息学院(IIITA));National Institute of Electronics & Information Technology (NIELIT)(国家电子与信息科技研究所(NIELIT))
Evaluating Retrieval-Augmented Generation Agents for Autonomous Scientific Discovery in Astrophysics
Institute of Astronomy, University of Cambridge, Cambridge, United Kingdom(英国剑桥大学天文研究所);Kavli Institute for Cosmology, University of Cambridge, Cambridge, United Kingdom(英国剑桥大学卡弗里宇宙研究所);Department of Physics, University of Cambridge, Cambridge, United Kingdom(英国剑桥大学物理系);Center for Computational Astrophysics, Flatiron Institute, New York, NY, USA(Flatiron研究所计算天体物理学中心);Department of Astrophysical Sciences, Princeton University, Princeton, NJ, USA(普林斯顿大学天体物理学系)
Evaluating Large Language Models in Scientific Discovery
Deep Principle(深原则);Department of Computer Science, Cornell University(计算机科学系,康奈尔大学);Department of Computer Science and Engineering, The Ohio State University(计算机科学与工程系,俄亥俄州立大学);Department of Chemical Engineering & Applied Chemistry, University of Toronto(化学工程与应用化学系,多伦多大学);Department of Computer Science and Engineering, University of Notre Dame(计算机科学与工程系,圣母大学);QuEra Computing Inc.(QuEra计算公司);Department of Pathology, Department of Genetics, Cancer Biology Program, Stanford University School of Medicine(病理学系、遗传学系、癌症生物学项目,斯坦福大学医学院);Harvard Law School(哈佛法学院);Department of Computer Science, Tufts University(计算机科学系,塔夫茨大学);School of Computational Science and Engineering, Georgia Institute of Technology(计算科学与工程学院,佐治亚理工学院);Department of Computer Science, University of California, Los Angeles(计算机科学系,加州大学洛杉矶分校);Department of Computer Science, Virginia Tech(计算机科学系,弗吉尼亚理工大学);Department of Physics, Tsinghua University(物理系,清华大学);Institute for Advanced Study, Tsinghua University(清华大学高级研究所);Laboratory of Artificial Chemical Intelligence, Ecole Polytechnique Federale de Lausanne(人工化学智能实验室,瑞士联邦理工学院)
Towards Biomarker Discovery for Early Cerebral Palsy Detection: Evaluating Explanations Through Kinematic Perturbations
Norwegian University of Science and Technology(挪威科技大学);Faculty of Medicine and Health Sciences(医学与健康科学学院);Faculty of Information Technology and Electrical Engineering(信息技术与电气工程学院);Faculty of Medicine(医学院)
A neural operator framework for data-driven discovery of stability and receptivity in physical systems
School of Computation, Information and Technology, Technical University of Munich(技术大学慕尼黑计算、信息与技术学院);Beijing Institute of Astronautical Systems Engineering(北京航天系统工程研究院)