Medical concept extraction from electronic health records underpins many downstream applications, yet remains challenging because medically meaningful concepts are frequently implied rather than explicitly stated in medical narratives. Existing benchmarks with human-annotated evidence spans underscore the importance of grounding extracted concepts in medical text. However, they predominantly focus on explicitly stated concepts instead of implicit concepts. We present MedicalBench, a benchmark for medical concept extraction with evidence grounding that evaluates implicit medical reasoning. MedicalBench formulates medical concept extraction as a verification task over medical note-concept pairs, coupled with sentence-level evidence identification. Built from MIMIC-IV discharge summaries and human-verified ICD-10 codes, the dataset is curated through a multi-stage large language model (LLM) triage pipeline followed by medical annotation and expert review. It deliberately includes implicit positives, semantically confusable negatives, and cases where LLM judgments disagree with medical expert assessments. We define two complementary evaluation tasks: (1) medical concept extraction and (2) sentence-level evidence retrieval, enabling assessment of both correctness and interpretability. Benchmarking state-of-the-art LLMs reveals that performance remains modest, highlighting the difficulty of extracting implicitly expressed concepts. We further show that performance is largely invariant to note length, indicating that MedicalBench isolates reasoning difficulty rather than superficial confounders. MedicalBench provides the first systematic benchmark for implicit, evidence-grounded medical concept extraction, offering a foundation for developing medical language models that can both identify medically relevant concepts and justify their predictions in a transparent and medically faithful manner.
Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Northeastern University（东北大学）；MD Anderson Cancer Center（MD安德森癌症中心）；Georgia Institute of Technology（佐治亚理工学院）
Evaluating large language models in medical applications: a survey
The Hong Kong Polytechnic University（香港理工大学）；Research Centre for SHARP Vision (RCSV), The Hong Kong Polytechnic University（香港理工大学SHARP视觉研究中心）；Centre for Eye and Vision Research (CEVR)（眼与视觉研究中心）；Renji Hospital, School of Medicine, Shanghai Jiao Tong University（上海交通大学医学院附属仁济医院）；Perception Vision Medical Technologies Co. Ltd.（感知视觉医疗科技有限公司）；The Hong Kong University of Science and Technology（香港科技大学）
Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgments
University of Georgia（佐治亚大学）；University of California, Merced（加州大学默塞德分校）；Lehigh University（莱特大学）；The University of Queensland（昆士兰大学）；National University of Singapore（新加坡国立大学）；University of California, Los Angeles（加州大学洛杉矶分校）
JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Models
Kyoto University（京都大学）；Hohai University（河海大学）；The University of Tokyo（东京大学）；University of Science and Technology of China（中国科学技术大学）；Hong Kong Polytechnic University（香港理工大学）
MedCalc-Bench: Evaluating Large Language Models for Medical Calculations
National Library of Medicine, National Institutes of Health（美国国立卫生研究院国家医学图书馆）；University of Virginia（弗吉尼亚大学）；University of Illinois at Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校）；Lapis Labs（拉皮斯实验室）；Yale University School of Medicine（耶鲁大学医学院）；Rosalind Franklin University Chicago Medical School（罗莎琳德·富兰克林大学芝加哥医学院）；Howard University College of Medicine（霍华德大学医学院）；University of Illinois College of Medicine at Chicago（芝加哥伊利诺伊大学医学院）；University of Chicago Pritzker School of Medicine（芝加哥大学普利兹克医学院）
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
East China Normal University（华东师范大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Hasso Plattner Institute（哈索·普拉特纳研究所）；University of Potsdam（波茨坦大学）；Cooperative Medianet Innovation Center, Shanghai Jiao Tong University（上海交通大学合作媒体网络创新中心）
Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving
Department of Electronic Engineering, Tsinghua University, Beijing, China（清华大学电子工程系）；College of AI, Tsinghua University, Beijing, China（清华大学人工智能学院）；School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China（北京邮电大学计算机科学学院）；iFLYTEK Research, Hefei, China（iFLYTEK研究院）
Evaluating multiple large language models in pediatric ophthalmology
Mayo Clinic（梅奥诊所）；the First Affiliated Hospital of Nanchang University（南昌大学第一附属医院）；Eye& ENT Hospital of Fudan University（复旦大学附属眼耳鼻喉科医院）；University of Georgia（佐治亚大学）；University of Electronic Science and Technology of China（电子科技大学）
MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Institute of Infectious Disease and Biosecurity, Fudan University（复旦大学上海传染病与生物安全研究院）；Shanghai Health Development Research Center (Shanghai Medical Information Center)（上海市卫生和健康发展研究中心（上海市医学科学技术情报研究所））；Imperial College London（伦敦帝国理工学院）
Evaluating Large Language Models on the Spanish Medical Intern Resident (MIR) Examination 2024/2025:A Comparative Analysis of Clinical Reasoning and Knowledge Application
Large Language Models Lack Temporal Awareness of Medical Knowledge
University of Virginia（弗吉尼亚大学）；National Institutes of Health（美国国家卫生研究院）；Dana-Farber Cancer Institute（达纳-法伯癌症研究所）；Yale University（耶鲁大学）；Weill Cornell Medicine（韦氏 Cornell 医学院）
PersianMedQA: Evaluating Large Language Models on a Persian-English Bilingual Medical Question Answering Benchmark
School of Electrical and Computer Engineering, University of Tehran（德黑兰大学电气与计算机工程学院）；Shahid Beheshti University of Medical Sciences（沙希德·贝赫什提医学科学大学）；Institute for Research in Fundamental Sciences (IPM)（基础科学研究所（IPM））
DrugBench: Evaluating AI Control Protocols for Medication Harm Mitigation
Universidad de Buenos Aires, Facultad de Ciencias Exactas y Naturales, Departamento de Computación（布宜诺斯艾利斯大学，精确与自然科学学院，计算机系）；AI Safety Argentina (AISAR)（阿根廷人工智能安全组织 (AISAR)）；Department of Computer Science, University of Oxford（牛津大学计算机科学系）；CONICET-Universidad de Buenos Aires, Instituto de Ciencias de la Computación (ICC)（阿根廷国家科学与技术研究理事会-布宜诺斯艾利斯大学，计算机科学研究所 (ICC)）
Mayo Clinic（梅奥诊所）；the First Affiliated Hospital of Nanchang University（南昌大学第一附属医院）；University of Georgia（佐治亚大学）；University of Electronic Science and Technology of China（电子科技大学）；Eye Institute of Xiamen University（厦门大学眼科研究所）
ECG-Expert-QA: A Benchmark for Evaluating Medical Large Language Models in Heart Disease Diagnosis
Zhejiang University School of Medicine（浙江大学医学院）；The First Affiliated Hospital（第一附属医院）；Liangzhu Laboratory（良渚实验室）；Shandong Jianzhu University（山东建筑大学）；FUXI AI Lab（伏羲人工智能实验室）；Beijing Normal University（北京师范大学）；Southern University of Science and Technology（南方科技大学）；China University of Geosciences (Wuhan)（中国地质大学（武汉））；Hangzhou Baorun Biotechnology Co., Ltd.（杭州宝润生物科技有限公司）
ClinBench-HPB: A Clinical Benchmark for Evaluating LLMs in Hepato-Pancreato-Biliary Diseases
The Hong Kong Polytechnic University（香港理工大学）；Shenzhen Institutes of Advanced Technology, CAS（深圳先进技术研究院）；Zhujiang Hospital, Southern Medical University（南方医科大学珠江医院）；The Key Laboratory of Biomedical Imaging Science and System, CAS（生物医学成像科学与系统重点实验室）
KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations
Kyoto University（京都大学）；The Hong Kong University of Science and Technology（香港科技大学）；Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学）；The University of Tokyo（东京大学）
Evaluating LLM-Based Translation of a Low-Resource Technical Language: The Medical and Philosophical Greek of Galen
Department of Classical and Mediterranean Studies, Vanderbilt University（维斯尼尔大学古典与地中海研究系）；Center for Hellenic Studies, Harvard University（哈佛大学希腊研究中心）；Department of Philosophy, Vanderbilt University（维斯尼尔大学哲学系）；Machine Intelligence and Normative Theory (MINT) Lab, Australian National University（澳大利亚国立大学机器智能与规范理论实验室）；Department of the Classics, Harvard University（哈佛大学古典学系）
