arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.22767cs.AIcs.CL
收藏

大语言模型代理能否生成真实世界证据?评估医学数据库中的观察性研究

Can LLM Agents Generate Real-World Evidence? Evaluating Observational Studies in Medical Databases

Zhejiang University(浙江大学)

Dubai Li, Yuxiang He, Yan Hu, Yu Tian, Jingsong Li

AI总结:

本研究评估了大语言模型代理在生成真实世界证据方面的能力,通过RWE-bench基准测试,发现代理在生成完整证据包方面存在显著局限,需进一步改进验证方法。

摘要:

观察性研究可以大规模生成临床可操作的证据,但在真实世界数据库中执行时具有开放性,需要在队列构建、分析和报告之间做出一致决策。以往对LLM代理的评估侧重于孤立步骤或单一答案,忽略了结果证据包的完整性和内部结构。为此,我们引入了RWE-bench基准,基于MIMIC-IV和同行评审的观察性研究衍生而来。每个任务提供相应研究协议作为参考标准,要求代理在真实数据库中执行实验并迭代生成树状结构的证据包。我们评估了六个LLM(三个开源,三个闭源)在三种代理框架下的表现,使用问题级别正确性和端到端任务指标。在162个任务中,任务成功率较低:最佳代理达到39.9%,最佳开源模型达到30.4%。代理框架也对性能指标有显著影响,导致超过30%的性能差异。此外,我们实现了自动化队列评估方法,以快速定位错误并识别代理失败模式。总体而言,结果突显了代理在生成完整证据包方面的持续局限,高效验证仍然是未来工作的重要方向。代码和数据可在https://github.com/somewordstoolate/RWE-bench获取。

英文摘要:

Observational studies can yield clinically actionable evidence at scale, but executing them on real-world databases is open-ended and requires coherent decisions across cohort construction, analysis, and reporting. Prior evaluations of LLM agents emphasize isolated steps or single answers, missing the integrity and internal structure of the resulting evidence bundle. To address this gap, we introduce RWE-bench, a benchmark grounded in MIMIC-IV and derived from peer-reviewed observational studies. Each task provides the corresponding study protocol as the reference standard, requiring agents to execute experiments in a real database and iteratively generate tree-structured evidence bundles. We evaluate six LLMs (three open-source, three closed-source) under three agent scaffolds using both question-level correctness and end-to-end task metrics. Across 162 tasks, task success is low: the best agent reaches 39.9%, and the best open-source model reaches 30.4%. Agent scaffolds also matter substantially, causing over 30% variation in performance metrics. Furthermore, we implement an automated cohort evaluation method to rapidly localize errors and identify agent failure modes. Overall, the results highlight persistent limitations in agents' ability to produce end-to-end evidence bundles, and efficient validation remains an important direction for future work. Code and data are available at https://github.com/somewordstoolate/RWE-bench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑