Anil Palepu, Vikram Dhillon, Polly Niravath, Wei-Hung Weng, Preethi Prasad, Khaled Saab, Ryutaro Tanno, Yong Cheng, Hanh Mai, Ethan Burns, Zainub Ajmal, Kavita Kulkarni, Philip Mansfield, Dale Webster, Joelle Barral, Juraj Gottweis, Mike Schaekermann, S. Sara Mahdavi, Vivek Natarajan, Alan Karthikesalingam, Tao Tu
Large language models (LLMs) have shown remarkable progress in encoding clinical knowledge and responding to complex medical queries with appropriate clinical reasoning. However, their applicability in subspecialist or complex medical settings remains underexplored. In this work, we probe the performance of AMIE, a research conversational diagnostic AI system, in the subspecialist domain of breast oncology care without specific fine-tuning to this challenging domain. To perform this evaluation, we curated a set of 50 synthetic breast cancer vignettes representing a range of treatment-naive and treatment-refractory cases and mirroring the key information available to a multidisciplinary tumor board for decision-making (openly released with this work). We developed a detailed clinical rubric for evaluating management plans, including axes such as the quality of case summarization, safety of the proposed care plan, and recommendations for chemotherapy, radiotherapy, surgery and hormonal therapy. To improve performance, we enhanced AMIE with the inference-time ability to perform web search retrieval to gather relevant and up-to-date clinical knowledge and refine its responses with a multi-stage self-critique pipeline. We compare response quality of AMIE with internal medicine trainees, oncology fellows, and general oncology attendings under both automated and specialist clinician evaluations. In our evaluations, AMIE outperformed trainees and fellows demonstrating the potential of the system in this challenging and important domain. We further demonstrate through qualitative examples, how systems such as AMIE might facilitate conversational interactions to assist clinicians in their decision making. However, AMIE's performance was overall inferior to attending oncologists suggesting that further research is needed prior to consideration of prospective uses.
OncoGPT: A Medical Conversational Model Tailored with Oncology Domain Expertise on a Large Language Model Meta-AI (LLaMA)
Shenzhen Kanghua Juntai Biotech Co. Ltd.（深圳康华君泰生物科技有限公司）；University of California, Davis（加利福尼亚大学戴维斯分校）；The First Affiliated Hospital of Jinan University（暨南大学附属第一医院）
CORAL: Expert-Curated medical Oncology Reports to Advance Language Model Inference
Bakar Computational Health Sciences Institute, University of California, San Francisco（加州大学旧金山分校巴卡尔计算健康科学研究所）；Helen Diller Family Comprehensive Cancer Center, University of California, San Francisco（加州大学旧金山分校海伦·迪勒家族综合癌症中心）；Center for Data-driven Insights and Innovation, University of California, Office of the President（加州大学校长办公室数据驱动洞察与创新中心）；Department of Pediatrics, University of California, San Francisco（加州大学旧金山分校）
Benchmarking ChatGPT-4 on ACR Radiation Oncology In-Training (TXIT) Exam and Red Journal Gray Zone Cases: Potentials and Challenges for AI-Assisted Medical Education and Decision Making in Radiation Oncology
University Hospital Erlangen（埃尔兰根大学医院）；Comprehensive Cancer Center Erlangen-EMN（埃尔兰根-EMN综合癌症中心）；Pattern Recognition Lab（模式识别实验室）
Ten Headache Specialists versus Artificial Intelligence for Clinical Literature Summarization: A Critical Evaluation and Comparison
Stanford University Palo Alto CA USA（斯坦福大学）；Department of Neurology Mayo Clinic Rochester MN USA（梅奥诊所神经科）；Department of Neurology Dalhousie University Halifax Canada（达尔豪斯大学神经科）；Jefferson Headache Center Department of Neurology Thomas Jefferson University PA USA（泰勒大学神经科）；Beth Israel Deaconess Medical Center Boston MA USA（贝斯以色列医疗中心）；Department of Neurology University of Florida Gainesville FL USA（佛罗里达大学神经科）；University of Colorado School of Medicine Department of Pediatrics Division of Child Neurology Aurora CO USA（科罗拉多医学院儿科部儿童神经科）；Department of Medicine Mount Sinai Hospital Icahn School of Medicine at Mount Sinai New York NY USA（西奈医院医学部）；Department of Neurology Mayo Clinic Scottsdale AZ USA（梅奥诊所Scottsdale分部）；Harvard Medical School Boston MA USA（哈佛医学院）；Department of Neurology Mount Sinai Hospital Icahn School of Medicine at Mount Sinai New York NY USA（西奈医院神经科）
RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Care
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Macau University of Science and Technology（澳门科技大学）；First Affiliated Hospital of Guangzhou Medical University（广州医科大学附属第一医院）；Guangzhou Institute of Respiratory Health（广州呼吸健康研究院）
Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models
National Hospital Organization Kagoshima Medical Center（国立医院机构鹿儿岛医疗中心）；Imamura General Hospital（今村综合医院）；Iwamoto Neurosurgery（岩本神经外科）；Kagoshima University（鹿儿岛大学）
Situated Action in Pre-Hospital Critical Care Dispatch: Identifying where and how Algorithmic Assistance might be useful in the daily work of specialist Emergency Medical Dispatchers
Swansea University（斯旺西大学）；EMRTS, Swansea Bay UHB（EMRTS，斯旺西湾大学健康保健董事会）
Development of a Large Language Model-based Multi-Agent Clinical Decision Support System for Korean Triage and Acuity Scale (KTAS)-Based Triage and Treatment Planning in Emergency Departments
John Bapst Memorial High School（约翰·巴普斯特纪念高中）；Minerva University（密涅瓦大学）
End-To-End Clinical Trial Matching with Large Language Models
National Center for Tumor Diseases (NCT), Heidelberg University Hospital（海德堡大学医院国家肿瘤疾病中心（NCT））；Else Kroener Fresenius Center for Digital Health, Technical University Dresden（德累斯顿工业大学Else Kroener Fresenius数字健康中心）；Heidelberg University（海德堡大学）；University Hospital Dresden（德累斯顿大学医院）；University Hospital RWTH Aachen（亚琛工业大学附属医学院）；University of St Andrews（圣安德鲁斯大学）；Technical University of Munich (TUM)（慕尼黑工业大学（TUM））；Google Cloud（谷歌云）；National Center for Tumor Diseases (NCT/UCC), Dresden（德累斯顿国家肿瘤疾病中心（NCT/UCC））；German Cancer Research Center (DKFZ)（德国癌症研究中心（DKFZ））；TUD Dresden University of Technology（德累斯顿工业大学（TUD））；Helmholtz-Zentrum Dresden-Rossendorf (HZDR)（德累斯顿-罗森多夫亥姆霍兹中心（HZDR））；Gustave Roussy（古斯塔夫·鲁西癌症中心）；Paris Saclay University（巴黎-萨克雷大学）；University Hospital Aache（亚琛大学医院）
Clinical Reasoning AI for Oncology Treatment Planning: A Multi-Specialty Case-Based Evaluation
Moffitt Cancer Center and Research Institute（莫菲特癌症中心与研究学院）；Innova Montréal Inc.（蒙特利尔创新公司）；Oncobrain, Inc.（Oncobrain公司）；Advanced Cancer Treatment Centers（先进癌症治疗中心）
Classifying Cancer Stage with Open-Source Clinical Large Language Models
1 College of Computing；Informatics Drexel University, Philadelphia PA, USA Email: cc3859,mml367；2 Department of Medical Oncology, Sidney Kimmel Cancer Center Thomas Jefferson University, Philadelphia, PA, USA Email
Conversational Agents in Behavioral Sleep Medicine: Designing Self-Report and Analytics Tools
The Johns Hopkins University（约翰斯·霍普金斯大学）；Department of Psychiatry and Behavioral Sciences, Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院精神病学与行为科学系）
Large Language Models in Peer-Run Community Behavioral Health Services: Understanding Peer Specialists and Service Users' Perspectives on Opportunities, Risks, and Mitigation Strategies
School of Computer Science（计算机科学系）；Carnegie Mellon University（卡内基梅隆大学）；Human-Computer Interaction Institute（人机交互研究所）；University of Pittsburgh（匹兹堡大学）；Graduate School of Applied and Professional Psychology（应用与专业心理学研究生院）；Rutgers University（罗格斯大学）；Software and Societal Systems Department（软件与社会系统部门）
AmarDoctor: An AI-Driven, Multilingual, Voice-Interactive Digital Health Application for Primary Care Triage and Patient Management to Bridge the Digital Health Divide for Bengali Speakers
MedAi Bangladesh Limited（MedAi孟加拉有限公司）；Central Police Hospital, Dhaka, Bangladesh（达卡警察医院）；University of Birmingham, UK（英国伯明翰大学）
