The integration of Large Language Models (LLMs) into the healthcare domain has the potential to significantly enhance patient care and support through the development of empathetic, patient-facing chatbots. This study investigates an intriguing question Can ChatGPT respond with a greater degree of empathy than those typically offered by physicians? To answer this question, we collect a de-identified dataset of patient messages and physician responses from Mayo Clinic and generate alternative replies using ChatGPT. Our analyses incorporate novel empathy ranking evaluation (EMRank) involving both automated metrics and human assessments to gauge the empathy level of responses. Our findings indicate that LLM-powered chatbots have the potential to surpass human physicians in delivering empathetic communication, suggesting a promising avenue for enhancing patient care and reducing professional burnout. The study not only highlights the importance of empathy in patient interactions but also proposes a set of effective automatic empathy ranking metrics, paving the way for the broader adoption of LLMs in healthcare.
Can "AI" Be a Doctor? A Study of Empathy, Readability, and Alignment in Clinical LLMs
Department of Electrical Engineering and Information Technology（电子工程与信息科技系）；University of Naples Federico II（那不勒斯费德里科二世大学）；Department of Translational Medical Sciences（转化医学科学系）；University of Campania ”Luigi Vanvitelli”（坎帕尼亚“路易吉·范维蒂利”大学）；Department of Computer Science, McCormick School of Engineering and Applied Science（计算机科学系，麦科马克工程与应用科学学院）；Northwestern University（西北大学）
The Biased Oracle: Assessing LLMs' Understandability and Empathy in Medical Diagnoses
Department of Chemistry and Applied Biosciences, ETH Zurich（应用生物科学系，苏黎世联邦理工学院）；Department of Computer Science, ETH Zurich（计算机科学系，苏黎世联邦理工学院）；Department of Health Sciences and Technology, ETH Zurich（健康科学与技术系，苏黎世联邦理工学院）
Assessing the Quality of AI-Generated Clinical Notes: A Validated Evaluation of a Large Language Model Scribe
Suki AI；Santa Clara Valley Medical Center（圣克拉拉谷医疗中心）；Stanford University School of Medicine（斯坦福大学医学院）；Stanford Cardiovascular Institute（斯坦福心血管研究所）；Hippocratic AI；The Permanente Medical Group（永久医疗集团）
Harvard Medical School（哈佛医学院）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Stanford University（斯坦福大学）；University of Alberta（阿尔伯塔大学）；Massachusetts General Hospital（麻省总医院）；Erasmus Medical Center（伊拉斯姆斯医学中心）；University of Maryland School of Medicine（马里兰大学医学院）；University of Maryland Institute for Health Computing（马里兰大学健康计算研究所）；VA Maryland Healthcare System（弗吉尼亚州马里兰医疗系统）；Brigham and Women’s Hospital（布莱根妇女医院）
Beyond Empathy: Integrating Diagnostic and Therapeutic Reasoning with Large Language Models for Mental Health Counseling
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济发展实验室）；SKL-IOTSC, CIS, University of Macau（澳门科学馆物联网与智慧城市研究中心）；Shandong University（山东大学）；Auckland University of Technology（技术大学（奥克兰））；University of Electronic Science and Technology of China（电子科技大学）
Rational Sensibility: LLM Enhanced Empathetic Response Generation Guided by Self-presentation Theory
Shenyang Institute of Computing Technology, Chinese Academy of Sciences（中国科学院沈阳计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Wenge Technology Co., Ltd（北京文阁科技有限公司）
EyeGPT: Ophthalmic Assistant with Large Language Models
The Hong Kong Polytechnic University（香港理工大学）；Centre for Eye and Vision Research (CEVR)（眼科与视觉研究中心）；Sun Yat-sen University（中山大学）；Shanghai General Hospital（上海市第一人民医院）；Shanghai Jiao Tong University（上海交通大学）
The Dialogue That Heals: A Comprehensive Evaluation of Doctor Agents' Inquiry Capability
Department of Computer Science and Technology, Tsinghua University, China（清华大学计算机科学与技术系）；Institute for AI Industry Research (AIR), Tsinghua University, China（清华大学人工智能产业研究院）
Conversation AI Dialog for Medicare powered by Finetuning and Retrieval Augmented Generation
University of Florida（佛罗里达大学）；National Institute of Technology Calicut（卡利卡特国家理工学院）；SRM, Chennai（SRM大学（金奈））；University of Cincinnati（辛辛那提大学）；Dartmouth College（达特茅斯学院）；Arizona State University（亚利桑那州立大学）
CASTLE: A Comprehensive Benchmark for Evaluating Student-Tailored Personalized Safety in Large Language Models
East China Normal University（华东师范大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Zhejiang University（浙江大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Education University of Hong Kong（香港教育大学）
ClinDEF: A Dynamic Evaluation Framework for Large Language Models in Clinical Reasoning
ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University（浙江大学杭州全球科技创新中心）；ZJU-UIUC Institute, Zhejiang University（浙江大学UIUC研究院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；The Polytechnic Institute, Zhejiang University（浙江大学 polytechnic 院）；AntGroup（蚂蚁集团）
AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows
School of Control Science and Engineering, Shandong University（控制科学与工程学院，山东大学）；Key Laboratory of Machine Intelligence and System Control, Shandong University（机器智能与系统控制重点实验室，山东大学）；Department of Medicine and Therapeutics, The Chinese University of Hong Kong（医学与治疗学系，香港中文大学）；Department of Geriatric Medicine, Qilu Hospital of Shandong University（老年医学科，山东大学齐鲁医院）；Department of Psychiatry, The Chinese University of Hong Kong（精神病学系，香港中文大学）；Li Chiu Kong Family Sleep Assessment Unit, Department of Psychiatry, Faculty of Medicine, The Chinese University of Hong Kong（李秋虹家庭睡眠评估单元，精神病学系，医学院，香港中文大学）；Li Ka Shing Institute of Health Sciences, Faculty of Medicine, The Chinese University of Hong Kong（李嘉诚健康科学研究院，医学院，香港中文大学）；Gerald Choa Neuroscience Institute, Department of Medicine and Therapeutics, The Chinese University of Hong Kong（Gerald Choa 神经科学研究所，医学与治疗学系，香港中文大学）
NYU Center for Data Science（纽约大学数据科学中心）；NYU Langone Health（纽约大学朗格尼医学中心）；NYU Tandon School of Engineering（纽约大学坦登工程学院）；NYU Grossman School of Medicine（纽约大学格罗斯曼医学院）；Prescient Design（Prescient设计公司）
Traditional evaluation metrics for classification in natural language processing such as accuracy and area under the curve fail to differentiate between models with different predictive behaviors despite their similar performance metrics. We introduce sensitivity score, a metric that scrutinizes models' behaviors at the vocabulary level to provide insights into disparities in their decision-making logic. We assess the sensitivity score on a set of representative words in the test set using two classifiers trained for hospital readmission classification with similar performance statistics. Our …
