arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.21074cs.AIq-bio.GNq-bio.QM
收藏

OmicsBench:面向AI智能体的研究级生物信息学长周期多轮基准测试

OmicsBench: a long-horizon multi-turn benchmark of AI agents on research-study-scale bioinformatics

Gideon Shaked, Siddharth Nagisetty

AI总结:

本文提出OmicsBench这一长周期多轮生物信息学基准测试,评估12种模型在三种工具上的表现,发现多轮评估能揭示单次测试未发现的智能体行为,还引入无监督流程分析失败模式。

摘要:

计算生物学家越来越多地在长周期多轮会话中将大量分析工作委托给AI智能体,但大多数用于评估智能体生物信息学能力的基准测试仅设置单个短任务和单次提示。本文提出OmicsBench,这是一个面向AI智能体的长周期多轮生物信息学基准测试。该基准测试的97个任务每个都将依赖子任务链接为有向无环图,总计包含662个基础子任务;每个子任务作为一轮用户提问,并根据答案密钥进行确定性评分,还可提供针对错误答案的上下文反馈。在三种测试工具上对12种模型进行测试后,最慢的智能体平均每个任务耗时3.6小时,且没有任何组合能接近该基准测试的饱和状态:无上下文反馈时平均得分为0.43至0.57(满分1),有上下文反馈时为0.54至0.67。多轮评估还揭示了单次提示无法发现的行为:智能体在依赖早期子任务的子任务上得分更低,而上下文反馈有助于智能体挽回部分损失。由于确定性评分无法定性说明智能体的失败原因,本文还引入了无监督失败模式发现流程,该流程对智能体的失败尝试进行描述、嵌入和聚类,以揭示新兴失败模式;将其应用于OmicsBench后,发现了智能体在开展生物信息学分析时的定性缺陷,包括智能体应用自身的统计过滤器替代指定过滤器,以及智能体凭记忆编写完整答案等。

英文摘要:

Computational biologists increasingly delegate substantial analyses to AI agents in long-horizon, multi-turn sessions, yet most benchmarks of agents' bioinformatics capabilities pose a single short task with a single-shot prompt. We present OmicsBench, a long-horizon, multi-turn bioinformatics benchmark for AI agents. Each of its 97 tasks links dependent subtasks into a directed acyclic graph, for 662 underlying subtasks in all. Each subtask is posed as one user turn and graded deterministically against an answer key, with optional in-context feedback on incorrect answers. Across 12 models on three harnesses, the slowest agent averages 3.6 hours per task, and no combination comes near saturating the benchmark, with mean scores of 0.43 to 0.57 out of a possible 1 without in-context feedback and 0.54 to 0.67 with it. Multi-turn evaluation also uncovers behavior that single-shot prompts cannot, as agents score lower on subtasks that depend on earlier subtasks, and in-context feedback helps agents recover some of that loss. Because deterministic scoring gives no qualitative account of how an agent failed, we also introduce an unsupervised failure mode discovery pipeline that describes, embeds, and clusters agents' failed attempts to surface emergent failure modes. Applied to OmicsBench, this pipeline exposes qualitative shortcomings in how agents carry out bioinformatics analyses, ranging from agents applying their own statistical filters in place of the specified ones to agents writing whole answers from memory.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑