Instruction-tuned Large Language Models (LLMs) can perform a wide range of tasks given natural language instructions to do so, but they are sensitive to how such instructions are phrased. This issue is especially concerning in healthcare, as clinicians are unlikely to be experienced prompt engineers and the potential consequences of inaccurate outputs are heightened in this domain.
This raises a practical question: How robust are instruction-tuned LLMs to natural variations in the instructions provided for clinical NLP tasks? We collect prompts from medical doctors across a range of tasks and quantify the sensitivity of seven LLMs -- some general, others specialized -- to natural (i.e., non-adversarial) instruction phrasings. We find that performance varies substantially across all models, and that -- perhaps surprisingly -- domain-specific models explicitly trained on clinical data are especially brittle, compared to their general domain counterparts. Further, arbitrary phrasing differences can affect fairness, e.g., valid but distinct instructions for mortality prediction yield a range both in overall performance, and in terms of differences between demographic groups.
A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment
Microsoft Healthcare & Life Sciences(微软医疗与生命科学);Microsoft Research Montréal(微软研究院蒙特利尔);Mila, Université de Montréal(蒙特利尔大学米拉研究所);IKIM, University Hospital Essen(埃森大学医院人工智能医学研究所);Cancer Research Center Cologne Essen(科隆埃森癌症研究中心);German Cancer Consortium(德国癌症联盟);TU Dortmund(多特蒙德工业大学)
Countering Catastrophic Forgetting of Large Language Models for Better Instruction Following via Weight-Space Model Merging
University of Florida(佛罗里达大学);Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida(佛罗里达大学医学院健康结果与生物医学信息学系);Department of Engineering Education, Herbert Wertheim College of Engineering, University of Florida(佛罗里达大学赫伯特·韦特海姆工程学院工程教育系);Preston A. Wells, Jr. Center for Brain Tumor Therapy, Lillian S. Wells Department of Neurosurgery, University of Florida(佛罗里达大学莉莉安·S·威尔斯神经外科系普雷斯顿·A·威尔斯脑肿瘤治疗中心)
OphIn-500K: Curating Web-Scale Visual Instructions for Scaling Ophthalmic Multimodal Large Language Models
Arizona State University(亚利桑那州立大学);Clemson University(克莱姆森大学);Washington University in St. Louis(圣路易斯华盛顿大学);University of Notre Dame(诺特丹大学);Florida State University(佛罗里达州立大学);Rice University(里德大学);NVIDIA(英伟达);Mayo Clinic(梅奥诊所)
Retrieval-Augmented and Knowledge-Grounded Language Models for Faithful Clinical Medicine
University of Oxford(牛津大学);Peking University(北京大学);Yale University(耶鲁大学);Tencent(腾讯);Oxford-Suzhou Centre for Advanced Research(牛津-苏州高等研究院);Shanghai Jiao Tong University School of Medicine(上海交通大学医学院)
Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework
Mila - Quebec AI Institute(魁北克人工智能研究所);McGill University(麦吉尔大学);University of Waterloo(滑铁卢大学);Vector Institute(向量研究所);Université de Sherbrooke(舍布鲁克大学);Université de Montréal(蒙特利尔大学)
Using Large Language Models to Generate Clinical Trial Tables and Figures
Novartis Pharmaceuticals Corporation(诺华制药公司);School of Biomedical Informatics The University of Texas Health Science Center at Houston(德克萨斯大学休斯顿健康科学中心生物医学信息学院);Novartis Pharma AG(诺华制药公司);The University of Texas MD Anderson Cancer Center(德克萨斯大学 MD 安德森癌症中心)
MedReadCtrl: Personalizing medical text generation with readability-controlled instruction learning
Center for Healthcare Organization and Implementation Research(医疗组织与实施研究中心);VA Bedford Health Care(贝福德医疗中心);Miner School of Computer and Information Sciences(米纳尔计算机与信息科学学院);University of Massachusetts Medical School(马萨诸塞大学医学学院);School of Nursing(护理学院);Zuckerberg College of Health Sciences(泽克伯格健康科学学院)
Towards Evaluating and Building Versatile Large Language Models for Medicine
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);China Mobile Communications Group Co., Ltd.(中国移动通信集团有限公司);China Mobile Communications Group Shanghai Co., Ltd.(中国移动通信集团上海有限公司)
MEIT: Multimodal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation
The Ohio State University(俄亥俄州立大学);Imperial College London(伦敦帝国理工学院);The University of Hong Kong(香港大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)
Generating Diverse Training Samples for Relation Extraction with Large Language Models
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics(人工智能学院,南京航空航天大学);MIIT Key Laboratory of Pattern Analysis and Machine Intelligence(信息产业部模式分析与机器智能重点实验室);The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education(脑机智能技术重点实验室,教育部)
Reliable Multilingual Orthopedic Decision Support from Clinical Narratives: Language-Aware Adaptation and Verification-Guided Deferral
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Software, Nanjing University of Information Science and Technology (NUIST)(南京信息工程大学软件学院);Department of Orthopedic, Bahawal Victoria Hospital(巴哈瓦尔维多利亚医院骨科部门)
A Specialized Large Language Model for Clinical Reasoning and Diagnosis in Rare Diseases
Guangzhou Women and Children's Medical Center, Guangzhou Medical University(广州医科大学附属广州市妇女儿童医疗中心);Tianjin Medical University(天津医科大学);Guangzhou National Laboratory(广州国家实验室);Children's Hospital of Fudan University(复旦大学附属儿童医院)
Susceptibility of Large Language Models to User-Driven Factors in Medical Queries
Yonsei University College of Medicine(延世大学医学院);Yonsei University(延世大学);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);Oncosoft Inc.(Oncosoft公司)
University of Massachusetts, Amherst(马萨诸塞大学阿默斯特分校);Emory University(埃默里大学);University of Minnesota(明尼苏达大学);University of Massachusetts, Lowell(马萨诸塞大学洛厄尔分校);UMass Chan Medical School(UMass Chan医学学院)
Navigating the digital spectrum: Assessing political bias, stability, and downstream fairness in Large Language Models
Jožef Stefan Institute(约热·斯泰凡研究所);Jožef Stefan International Postgraduate School(约热·斯泰凡国际研究生学院);University of La Rochelle(拉罗谢尔大学);University of Ljubljana(卢布尔雅那大学)