Observational studies can yield clinically actionable evidence at scale, but executing them on real-world databases is open-ended and requires coherent decisions across cohort construction, analysis, and reporting. Prior evaluations of LLM agents emphasize isolated steps or single answers, missing the integrity and internal structure of the resulting evidence bundle. To address this gap, we introduce RWE-bench, a benchmark grounded in MIMIC-IV and derived from peer-reviewed observational studies. Each task provides the corresponding study protocol as the reference standard, requiring agents to execute experiments in a real database and iteratively generate tree-structured evidence bundles. We evaluate six LLMs (three open-source, three closed-source) under three agent scaffolds using both question-level correctness and end-to-end task metrics. Across 162 tasks, task success is low: the best agent reaches 39.9%, and the best open-source model reaches 30.4%. Agent scaffolds also matter substantially, causing over 30% variation in performance metrics. Furthermore, we implement an automated cohort evaluation method to rapidly localize errors and identify agent failure modes. Overall, the results highlight persistent limitations in agents' ability to produce end-to-end evidence bundles, and efficient validation remains an important direction for future work. Code and data are available at https://github.com/somewordstoolate/RWE-bench.
Evaluating large language models in medical applications: a survey
The Hong Kong Polytechnic University(香港理工大学);Research Centre for SHARP Vision (RCSV), The Hong Kong Polytechnic University(香港理工大学SHARP视觉研究中心);Centre for Eye and Vision Research (CEVR)(眼与视觉研究中心);Renji Hospital, School of Medicine, Shanghai Jiao Tong University(上海交通大学医学院附属仁济医院);Perception Vision Medical Technologies Co. Ltd.(感知视觉医疗科技有限公司);The Hong Kong University of Science and Technology(香港科技大学)
JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Models
Kyoto University(京都大学);Hohai University(河海大学);The University of Tokyo(东京大学);University of Science and Technology of China(中国科学技术大学);Hong Kong Polytechnic University(香港理工大学)
MedCalc-Bench: Evaluating Large Language Models for Medical Calculations
National Library of Medicine, National Institutes of Health(美国国立卫生研究院国家医学图书馆);University of Virginia(弗吉尼亚大学);University of Illinois at Urbana Champaign(伊利诺伊大学厄巴纳-香槟分校);Lapis Labs(拉皮斯实验室);Yale University School of Medicine(耶鲁大学医学院);Rosalind Franklin University Chicago Medical School(罗莎琳德·富兰克林大学芝加哥医学院);Howard University College of Medicine(霍华德大学医学院);University of Illinois College of Medicine at Chicago(芝加哥伊利诺伊大学医学院);University of Chicago Pritzker School of Medicine(芝加哥大学普利兹克医学院)
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
East China Normal University(华东师范大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Hasso Plattner Institute(哈索·普拉特纳研究所);University of Potsdam(波茨坦大学);Cooperative Medianet Innovation Center, Shanghai Jiao Tong University(上海交通大学合作媒体网络创新中心)
Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving
Department of Electronic Engineering, Tsinghua University, Beijing, China(清华大学电子工程系);College of AI, Tsinghua University, Beijing, China(清华大学人工智能学院);School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China(北京邮电大学计算机科学学院);iFLYTEK Research, Hefei, China(iFLYTEK研究院)
Evaluating Large Language Models on the Spanish Medical Intern Resident (MIR) Examination 2024/2025:A Comparative Analysis of Clinical Reasoning and Knowledge Application
This Treatment Works, Right? Evaluating LLM Sensitivity to Patient Question Framing in Medical QA
Northeastern University(东北大学);UC Berkeley(加州大学伯克利分校);UT Austin(德克萨斯大学奥斯汀分校);UT MD Anderson Cancer Center(德克萨斯大学MD安德森癌症中心);Georgia Institute of Technology(佐治亚理工学院)
Mayo Clinic(梅奥诊所);the First Affiliated Hospital of Nanchang University(南昌大学第一附属医院);University of Georgia(佐治亚大学);University of Electronic Science and Technology of China(电子科技大学);Eye Institute of Xiamen University(厦门大学眼科研究所)
Text-to-SQL Generation for Question Answering on Electronic Medical Records
Virginia Tech(弗吉尼亚理工大学)
电子医疗记录(EMR)包含全面的患者信息,并且通常存储在具有多个表的关系数据库中。从电子病历数据中有效地检索患者信息检索是医学专家的一项具有挑战性的任务。SQL到SQL生成方法通过首先预测关于数据库的给定问题的SQL查询,然后在数据库上执行查询来解决这个问题。然而,大多数现有的方法还没有适应医疗保健领域,由于缺乏医疗保健数据库到SQL数据集的学习模型特定于这个领域。此外,术语缩写的广泛使用和问题中可能的错别字为准确生成相应的SQL查询带来了额外的挑战。在本文中,我们通过开发一个基于深度学习的TRanslate-Edit Model for SQL(TREQS)生成来解决这些挑战,该模型采用广泛使用的序列到序列模型来直接生成给定问题的SQL查询,并使用注意力复制机制和特定于任务的查找表进一步执行所需的编辑。基于广泛使用的公共电子医疗数据库,我们创建了一个新的大规模的SQL对数据集,命名为MIMICSQL,以执行在医疗领域的SQL到SQL的生成任务。一组广泛的实验进行评估我们提出的模型MIMICSQL的性能。定量和定性的实验结果表明,我们所提出的方法在预测条件值的灵活性和效率,其鲁棒性随机问题的缩写和错别字。
Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgments
University of Georgia(佐治亚大学);University of California, Merced(加州大学默塞德分校);Lehigh University(莱特大学);The University of Queensland(昆士兰大学);National University of Singapore(新加坡国立大学);University of California, Los Angeles(加州大学洛杉矶分校)
Evaluating Fine-Tuned LLM Model For Medical Transcription With Small Low-Resource Languages Validated Dataset
School of ICT and Industrial Management(信息通信技术与工业管理学院);Metropolia University of Applied Sciences(Metropolia应用科学大学);Rensselaer Polytechnic Institute(伦斯勒理工学院)
Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence
The University of Texas at Austin(德克萨斯大学奥斯汀分校);Northeastern University(东北大学);MD Anderson Cancer Center(MD安德森癌症中心);Georgia Institute of Technology(佐治亚理工学院)
DrugBench: Evaluating AI Control Protocols for Medication Harm Mitigation
Universidad de Buenos Aires, Facultad de Ciencias Exactas y Naturales, Departamento de Computación(布宜诺斯艾利斯大学,精确与自然科学学院,计算机系);AI Safety Argentina (AISAR)(阿根廷人工智能安全组织 (AISAR));Department of Computer Science, University of Oxford(牛津大学计算机科学系);CONICET-Universidad de Buenos Aires, Instituto de Ciencias de la Computación (ICC)(阿根廷国家科学与技术研究理事会-布宜诺斯艾利斯大学,计算机科学研究所 (ICC))
KidneyTalk-open: No-code Deployment of a Private Large Language Model with Medical Documentation-Enhanced Knowledge Database for Kidney Disease
Tianjin University of Technology(天津理工大学);National Institute of Health Data Science, Peking University(北京大学健康数据科学国家研究院);Peking University First Hospital(北京大学第一医院);Peking University People’s Hospital(北京大学人民医院);Institute of Internet Industry, Tsinghua University(清华大学互联网产业研究院);Chinese Academy of Medical Sciences(中国医学科学院);State Key Laboratory of Vascular Homeostasis and Remodeling, Peking University(北京大学血管稳态与重构国家重点实验室)
Marking the Wrong Symptoms: Evaluating LLM Watermarks in Medical Texts
ETH Zurich(苏黎世联邦理工学院);Berlin Institute of Health at Charité – Universitätsmedizin Berlin(柏林夏里特医学院柏林健康研究所);Deutsches Herzzentrum der Charité – Medical Heart Center of Charité and German Heart Institute Berlin(柏林夏里特医学院德国心脏中心与柏林德国心脏研究所)
ClinBench-HPB: A Clinical Benchmark for Evaluating LLMs in Hepato-Pancreato-Biliary Diseases
The Hong Kong Polytechnic University(香港理工大学);Shenzhen Institutes of Advanced Technology, CAS(深圳先进技术研究院);Zhujiang Hospital, Southern Medical University(南方医科大学珠江医院);The Key Laboratory of Biomedical Imaging Science and System, CAS(生物医学成像科学与系统重点实验室)
ECG-Expert-QA: A Benchmark for Evaluating Medical Large Language Models in Heart Disease Diagnosis
Zhejiang University School of Medicine(浙江大学医学院);The First Affiliated Hospital(第一附属医院);Liangzhu Laboratory(良渚实验室);Shandong Jianzhu University(山东建筑大学);FUXI AI Lab(伏羲人工智能实验室);Beijing Normal University(北京师范大学);Southern University of Science and Technology(南方科技大学);China University of Geosciences (Wuhan)(中国地质大学(武汉));Hangzhou Baorun Biotechnology Co., Ltd.(杭州宝润生物科技有限公司)
Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning
Stratified Precision;Harvard Medical School(哈佛医学院);Imperial College London(帝国理工学院伦敦分校);National Health Service(国家健康服务系统);Ipsen France(Ipsen法国);University College London(伦敦大学学院)
PersianMedQA: Evaluating Large Language Models on a Persian-English Bilingual Medical Question Answering Benchmark
School of Electrical and Computer Engineering, University of Tehran(德黑兰大学电气与计算机工程学院);Shahid Beheshti University of Medical Sciences(沙希德·贝赫什提医学科学大学);Institute for Research in Fundamental Sciences (IPM)(基础科学研究所(IPM))
Large Language Models Lack Temporal Awareness of Medical Knowledge
University of Virginia(弗吉尼亚大学);National Institutes of Health(美国国家卫生研究院);Dana-Farber Cancer Institute(达纳-法伯癌症研究所);Yale University(耶鲁大学);Weill Cornell Medicine(韦氏 Cornell 医学院)