Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm
Ant Group(蚂蚁集团);Shanghai Jiao Tong University School of Medicine Affiliated Renji Hospital(上海交通大学医学院附属仁济医院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
Lei Liu, Xiaoyan Yang, Fangzhou Li, Chenfei Chi, Yue Shen, Shiwei Lyu Ming Zhang, Xiaowei Ma, Xiangguo Lyu, Liya Ma, Zhiqiang Zhang, Wei Xue, Yiran Huang, Jinjie Gu
Large language models (LLMs) are gaining increasing interests to improve clinical efficiency for medical diagnosis, owing to their unprecedented performance in modelling natural language. Ensuring the safe and reliable clinical applications, the evaluation of LLMs indeed becomes critical for better mitigating the potential risks, e.g., hallucinations. However, current evaluation methods heavily rely on labor-intensive human participation to achieve human-preferred judgements. To overcome this challenge, we propose an automatic evaluation paradigm tailored to assess the LLMs' capabilities in delivering clinical services, e.g., disease diagnosis and treatment. The evaluation paradigm contains three basic elements: metric, data, and algorithm. Specifically, inspired by professional clinical practice pathways, we formulate a LLM-specific clinical pathway (LCP) to define the clinical capabilities that a doctor agent should possess. Then, Standardized Patients (SPs) from the medical education are introduced as the guideline for collecting medical data for evaluation, which can well ensure the completeness of the evaluation procedure. Leveraging these steps, we develop a multi-agent framework to simulate the interactive environment between SPs and a doctor agent, which is equipped with a Retrieval-Augmented Evaluation (RAE) to determine whether the behaviors of a doctor agent are in accordance with LCP. The above paradigm can be extended to any similar clinical scenarios to automatically evaluate the LLMs' medical capabilities. Applying such paradigm, we construct an evaluation benchmark in the field of urology, including a LCP, a SPs dataset, and an automated RAE. Extensive experiments are conducted to demonstrate the effectiveness of the proposed approach, providing more insights for LLMs' safe and reliable deployments in clinical practice.
KREL: Automatic Medical Coding via Knowledge-Guided Reasoning over Clinical Evidence with LLMs
The University of New South Wales(新南威尔士大学);University of Göttingen(哥廷根大学);Macquarie University(麦考瑞大学);Beijing Intelligent Decision Medical Technology Co. Ltd(北京智决医疗科技有限公司)
CliniChat: A Multi-Source Knowledge-Driven Framework for Clinical Interview Dialogue Reconstruction and Evaluation
College of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);Zhongnan Hospital of Wuhan University(武汉大学中南医院);Renmin Hospital of Wuhan University(武汉大学人民医院)
MedMobile: A mobile-sized language model with clinical capabilities
NYU Langone Medical Center(纽约大学朗格尼医学中心);The University of Texas at Austin(德克萨斯大学奥斯汀分校);Washington University School of Medicine in St. Louis(圣路易斯华盛顿大学医学院);NYU Langone Health(纽约大学朗格尼健康中心);New York University(纽约大学)
Classifying Long Clinical Documents with Pre-trained Transformers
University of Arizona(亚利桑那大学);Boston Children’s Hospital(波士顿儿童医院);Harvard Medical School(哈佛医学院);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Loyola University Chicago(芝加哥洛约拉大学)
A Multi-agent Large Language Model Framework to Automatically Assess Performance of a Clinical AI Triage Tool
Thomas Jefferson University(托马斯杰斐逊大学);Weil-Cornell Medical Center(威尔康奈尔医学中心);The Ohio State University(俄亥俄州立大学);The Jackson Lab(杰克逊实验室);Unity Health(联合健康)
DeepSeek in Healthcare: A Survey of Capabilities, Risks, and Clinical Applications of Open-Source Large Language Models
Weill Cornell Medicine, Cornell University(威尔康奈尔医学院,康奈尔大学);Johns Hopkins University(约翰斯·霍普金斯大学);Touro University College of Osteopathic Medicine(图罗大学整骨医学院)
Enhancing Large Language Models for Clinical Decision Support by Incorporating Clinical Practice Guidelines
University of Pittsburgh(匹兹堡大学);University of Pittsburgh Medical Center(匹兹堡大学医学中心);University of Pittsburgh Medical Center Western Maryland(匹兹堡大学医学中心西马里兰分院)
WangLab at MEDIQA-Chat 2023: Clinical Note Generation from Doctor-Patient Conversations using Large Language Models
University of Toronto(多伦多大学);Terrence Donnelly Centre for Cellular and Biomolecular Research(特伦斯·唐纳利细胞与生物分子研究中心);Vector Institute for AI(向量人工智能研究院);University Health Network(大学健康网络);Sunnybrook Health Sciences Centre(桑尼布鲁克健康科学中心)
Detecting automatically the layout of clinical documents to enhance the performances of downstream natural language processing
Sorbonne Université(索邦大学);Inserm(法国国家健康与医学研究院);Institut Pierre-Louis d'Epidémiologie et de Santé Publique(皮埃尔-路易流行病学与公共卫生研究所);Assistance Publique-Hôpitaux de Paris(巴黎公共医疗集团);LIMICS(医学信息学与信号处理实验室)
Clinical Assistant Diagnosis for Electronic Medical Record Based on Convolutional Neural Network
Tsinghua University(清华大学);Tsinghua National Laboratory of Information Science and Technology(清华信息科学与技术国家实验室);Beijing University of Posts and Telecommunications(北京邮电大学);Huangshi Central Hospital of Edong Healthcare Group, Hubei Polytechnic University(湖北理工学院鄂东医疗集团黄石市中心医院)
Auxiliary Knowledge-Induced Learning for Automatic Multi-Label Medical Document Classification
University of Western Ontario(西安大略大学);Vector Institute for Artificial Intelligence(向量人工智能研究院);Dalhousie University(达尔豪斯大学);University of Toronto(多伦多大学)
SIFTING: A Novel LLM-Based Framework for Structured and Transparent Information Extraction from Clinical Free-Text Reports, with Application to Tumor Staging in Lung Cancer
Axana B.V.(阿克萨纳有限公司);University College London Hospitals NHS Foundation Trust(伦敦大学学院医院 NHS 基金会信托);University College London(伦敦大学学院);Mid & South Essex NHS Foundation Trust(中埃塞克斯及南埃塞克斯 NHS 基金会信托);NHS Greater Glasgow and Clyde(大格拉斯哥及克莱德 NHS)
ReEvalMed: Rethinking Medical Report Evaluation by Aligning Metrics with Real-World Clinical Judgment
Technical University of Munich(慕尼黑技术大学);Munich Center for Machine Learning(慕尼黑机器学习中心);University of Massachusetts Boston(马萨诸塞大学波士顿分校);University of Strasbourg(斯特拉斯堡大学)