As large language models (LLMs) develop increasingly sophisticated capabilities and find applications in medical settings, it becomes important to assess their medical safety due to their far-reaching implications for personal and public health, patient safety, and human rights. However, there is little to no understanding of the notion of medical safety in the context of LLMs, let alone how to evaluate and improve it. To address this gap, we first define the notion of medical safety in LLMs based on the Principles of Medical Ethics set forth by the American Medical Association. We then leverage this understanding to introduce MedSafetyBench, the first benchmark dataset designed to measure the medical safety of LLMs. We demonstrate the utility of MedSafetyBench by using it to evaluate and improve the medical safety of LLMs. Our results show that publicly-available medical LLMs do not meet standards of medical safety and that fine-tuning them using MedSafetyBench improves their medical safety while preserving their medical performance. By introducing this new benchmark dataset, our work enables a systematic study of the state of medical safety in LLMs and motivates future work in this area, paving the way to mitigate the safety risks of LLMs in medicine. The benchmark dataset and code are available at https://github.com/AI4LIFE-GROUP/med-safety-bench.
CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs
Grado Department of Industrial & Systems Engineering, Virginia Tech, Blacksburg, VA 24061, USA(弗吉尼亚理工大学格拉多工业与系统工程系,弗吉尼亚理工大学,布莱克斯堡,VA 24061,美国);Department of Computer Science, Virginia Tech, Blacksburg, VA 24061, USA(弗吉尼亚理工大学计算机科学系,弗吉尼亚理工大学,布莱克斯堡,VA 24061,美国);Dept of Industrial, Manufacturing, and Systems Engineering at University of Texas at Arlington, Arlington, TX 76019, USA(德克萨斯理工大学阿灵顿分校工业、制造与系统工程系,阿灵顿,TX 76019,美国)
The Forgotten Shield: Safety Grafting in Parameter-Space for Medical MLLMs
National University of Defense Technology(国防科技大学);Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所多模态人工智能系统(MAIS));Multimedia Department, Xiaomi Inc(小米公司多媒体部门);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong(香港科学院人工智能与机器人中心);School of Artificial Intelligence, University of Chinese Academy of Sciences, UCAS(中国科学院大学人工智能学院)
SafeMed-R1: Clinician-Audited Safety and Ethics Alignment for Medical Large Language Models
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Joint Laboratory of Biomedical Artificial Intelligence(生物医学人工智能联合实验室);Shanghai Institute of Infectious Disease and Biosecurity(上海传染病与生物安全研究院);Shanghai Health Development Research Center (Shanghai Medical Information Center)(上海健康发展战略研究中心(上海医疗信息中心));University of Washington(华盛顿大学);Department of Eye and Vision Sciences, University of Liverpool(利物浦大学眼科与视觉科学系);Liverpool Centre for Cardiovascular Science, University of Liverpool(利物浦大学心血管科学中心);School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院)
JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Models
Kyoto University(京都大学);Hohai University(河海大学);The University of Tokyo(东京大学);University of Science and Technology of China(中国科学技术大学);Hong Kong Polytechnic University(香港理工大学)
The Need for Guardrails with Large Language Models in Medical Safety-Critical Settings: An Artificial Intelligence Application in the Pharmacovigilance Ecosystem
Harvard-MIT Department of Health Sciences and Technology(哈佛-麻省理工卫生科学与技术部);GlaxoSmithKline(葛兰素史克公司);London School of Hygiene and Tropical Medicine(伦敦卫生与热带医学院);Harvard T.H. Chan School of Public Health(哈佛陈曾熙公共卫生学院)
Safety challenges of AI in medicine in the era of large language models
National University of Singapore(新加坡国立大学);University of Cambridge(剑桥大学);Mila-Quebec AI Institute(米拉-魁北克人工智能研究所);Harvard Medical School(哈佛医学院);Massachusetts General Hospital(麻省总医院);Yale University(耶鲁大学);Brigham and Women’s Hospital/Dana-Farber Cancer Institute(布莱根妇女医院/达纳-法伯癌症研究所);The Hong Kong University of Science and Technology(香港科技大学);Stanford University(斯坦福大学)
AI for Regulatory Affairs: Balancing Accuracy, Interpretability, and Computational Cost in Medical Device Classification
Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford(生物医学工程研究所,工程科学系,牛津大学);Birmingham Law School, University of Birmingham(伯明翰法学院,伯明翰大学)
Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine
University of Maryland at College Park(马里兰大学帕克分校);University of Virginia(弗吉尼亚大学);Howard University(霍华德大学);Albert Einstein College of Medicine(阿尔伯特·爱因斯坦医学院);Universidad de Chile(智利大学);University of Florida(佛罗里达大学)
MedForge: Interpretable Medical Deepfake Detection via Forgery-aware Reasoning
National University of Singapore(新加坡国立大学);The Chinese University of Hong Kong(香港中文大学);Hunan University(湖南大学);Xi’an Jiaotong University(西安交通大学);Guangdong Provincial People’s Hospital(广东省人民医院)
Reasoning LLMs in the Medical Domain: A Literature Survey
Fraunhofer IAIS - Department of Media Engineering(弗劳恩霍夫研究所媒体工程部门);University of Bonn - Department of Computer Science(波恩大学计算机科学系);West-AI - Federal Ministry of Education and Research(西德人工智能 - 教育与研究部)
Knowing When to Abstain: Medical LLMs Under Clinical Uncertainty
Manning College of Information and Computer Sciences, UMass Amherst, MA, USA(马萨诸塞大学阿姆赫斯特曼宁信息与计算机科学学院);Center for Healthcare Organization and Implementation Research, VA Bedford Health Care(医疗组织与实施研究中心);Miner School of Computer and Information Sciences, UMass Lowell, MA, USA(米纳尔计算机与信息科学学院)
Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models
Stanford University(斯坦福大学);Shenzhen University(深圳大学);The Chinese University of Hong Kong(香港中文大学);City University of Hong Kong(香港城市大学);Renmin University of China(中国人民大学)
MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine
School of Software Engineering, Sun Yat-sen University(中山大学软件学院);Institute of TCM Diagnostics, Hunan University of Chinese Medicine(湖南中医药大学中医诊断研究所);School of Artificial Intelligence, Guilin University of Electronic Technology(桂林电子科技大学人工智能学院);Department of Computer Science, Cornell University(康奈尔大学计算机科学系)
DrugBench: Evaluating AI Control Protocols for Medication Harm Mitigation
Universidad de Buenos Aires, Facultad de Ciencias Exactas y Naturales, Departamento de Computación(布宜诺斯艾利斯大学,精确与自然科学学院,计算机系);AI Safety Argentina (AISAR)(阿根廷人工智能安全组织 (AISAR));Department of Computer Science, University of Oxford(牛津大学计算机科学系);CONICET-Universidad de Buenos Aires, Instituto de Ciencias de la Computación (ICC)(阿根廷国家科学与技术研究理事会-布宜诺斯艾利斯大学,计算机科学研究所 (ICC))
MIRA: A Bilingual Benchmark for Medical Information Response Audit
The University of Chicago(芝加哥大学);SynAI Technologies Inc.(SynAI技术公司);Jinzhou Medical University(锦州医学院);Zhejiang University(浙江大学);Dartmouth College(达特茅斯学院);Northwestern University(西北大学)
MedSentry: Understanding and Mitigating Safety Risks in Medical LLM Multi-Agent Systems
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术全国重点实验室);School of Computer Science, Carnegie Mellon University(卡内基梅隆大学计算机科学学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);Department of Electrical and Electronic Engineering, The University of Hong Kong(香港大学电机电子工程系)
Ensuring Medical AI Safety: Interpretability-Driven Detection and Mitigation of Spurious Model Behavior and Associated Data
Fraunhofer Heinrich Hertz Institut(弗劳恩霍夫 Heinrich Hertz 研究所);Technische Universität Berlin(柏林技术大学);Berlin Institute for the Foundations of Learning and Data (BIFOLD)(柏林学习与数据基础研究所(BIFOLD))