arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.07718cs.AIcs.CVcs.LG
收藏

评估AI代理在神经科学数据到发现流程中的案例研究

A case study of evaluating AI agents on a neuroscience data-to-discovery pipeline

Cornell University(康奈尔大学);HHMI Janelia Research Campus(霍华德·休斯医学研究所贾雷尔研究园区)

Kai A. Horstmann, Ethan Lin, Alice A. Robie, Jennifer J. Sun, Kristin Branson

AI总结:

本研究评估通用编码代理在果蝇光遗传学数据到发现流程中的表现,发现代理能解决单个阶段任务,但端到端流程仍超出其能力,主要挑战包括缺乏预定义迭代标准和科学判断能力。

摘要:

代理型AI工具为自动化科学研究流程中的软件开发瓶颈提供了有希望的路径,特别是对于那些需要领域专家花费数天到数月构建的阶段,科学家关心的是正确性和鲁棒性,而非实现细节。我们针对果蝇光遗传学数据到发现流程,对通用编码代理进行了实证研究。我们在比现有基准大得多的任务、数量级更大的数据集以及基于领域专家标准的评估标准上评估代理。我们表明,代理可以解决几个单独的流程阶段,这表明阶段级自动化是可行的。通过分析代理的代码迭代,我们发现当没有预定义的标准可供迭代时,它们最困难,此时它们必须利用自己的科学判断来评估当前解决方案,这是一个关键开放挑战。与科学实践相呼应,它们有时尝试对中间输出进行视觉检查以进行自我评估,但大多未能正确解释所见或据此采取行动。正确解决端到端流程需要将所有流程阶段的成功串联起来,这超出了代理当前的能力。我们识别出现有基准中基本缺失的挑战,包括计算资源管理和对大型保留数据集的泛化。最后,我们提炼出构建科学任务和针对开放问题的严格评估标准的原则。

英文摘要:

Agentic AI offers a promising path to automating software development bottlenecks in scientific research pipelines, particularly for stages that take domain experts days to months to build and where correctness and robustness matter more than implementation details. We present an empirical study of general-purpose coding agents on a fly optogenetics data-to-discovery pipeline. We assess agents on tasks and datasets substantially larger than existing benchmarks and evaluation criteria grounded in domain expert standards. We show that agents can solve several pipeline stages, suggesting stage-level automation is tractable. By analyzing agents' code iterations, we show they struggle most without a pre-defined criterion, when they must instead use their scientific judgment to assess their current solution. Mirroring scientists, they sometimes attempt visual inspection of intermediate outputs for self-evaluation, but largely fail to interpret what they see or act on it appropriately. Solving the end-to-end pipeline requires stringing together successes across all stages, which is beyond agents' current abilities. We identify challenges largely absent from existing benchmarks, including computational resource management and generalization to held-out data. Finally, we distill principles for constructing scientific tasks and rigorous evaluation criteria for open-ended problems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑