从任务执行者到研究伙伴:通过工作流整合在生物医学研究中评估AI共 pilot
From Task Executors to Research Partners: Evaluating AI Co-Pilots Through Workflow Integration in Biomedical Research
AI总结:
本文提出了一种以过程为导向的评估框架,用于评估AI系统在生物医学研究中的协作能力,强调对话质量、工作流编排、会话连续性和研究人员体验等关键维度。
摘要:
人工智能系统越来越多地应用于生物医学研究。然而,当前的评估框架可能无法充分评估其作为研究合作者的有效性。这项快速综述检查了AI系统在临床前生物医学研究中的基准测试实践。从2018年1月1日到2025年10月31日,搜索了三个主要数据库和两个预印本服务器,确定了14个基准测试,这些测试评估了AI在文献理解、实验设计和假设生成方面的能力。结果表明,所有当前的基准测试都评估了孤立组件的能力,包括数据分析质量、假设有效性以及实验协议设计。然而,真正的研究合作需要跨多个会话的工作流,具有上下文记忆、适应性对话和约束传播。这种差距意味着在组件基准测试中表现优异的系统可能在作为实际研究共 pilot 时失败。提出了一种以过程为导向的评估框架,该框架解决了当前基准测试中缺失的四个关键维度:对话质量、工作流编排、会话连续性和研究人员体验。这些维度对于评估AI系统作为研究共 pilot 而不是孤立任务执行者至关重要。
英文摘要:
Artificial intelligence systems are increasingly deployed in biomedical research. However, current evaluation frameworks may inadequately assess their effectiveness as research collaborators. This rapid review examines benchmarking practices for AI systems in preclinical biomedical research. Three major databases and two preprint servers were searched from January 1, 2018 to October 31, 2025, identifying 14 benchmarks that assess AI capabilities in literature understanding, experimental design, and hypothesis generation. The results revealed that all current benchmarks assess isolated component capabilities, including data analysis quality, hypothesis validity, and experimental protocol design. However, authentic research collaboration requires integrated workflows spanning multiple sessions, with contextual memory, adaptive dialogue, and constraint propagation. This gap implies that systems excelling on component benchmarks may fail as practical research co-pilots. A process-oriented evaluation framework is proposed that addresses four critical dimensions absent from current benchmarks: dialogue quality, workflow orchestration, session continuity, and researcher experience. These dimensions are essential for evaluating AI systems as research co-pilots rather than as isolated task executors.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。