Large Language Models (LLMs) have demonstrated remarkable proficiency in human interactions, yet their application within the medical field remains insufficiently explored. Previous works mainly focus on the performance of medical knowledge with examinations, which is far from the realistic scenarios, falling short in assessing the abilities of LLMs on clinical tasks. In the quest to enhance the application of Large Language Models (LLMs) in healthcare, this paper introduces the Automated Interactive Evaluation (AIE) framework and the State-Aware Patient Simulator (SAPS), targeting the gap between traditional LLM evaluations and the nuanced demands of clinical practice. Unlike prior methods that rely on static medical knowledge assessments, AIE and SAPS provide a dynamic, realistic platform for assessing LLMs through multi-turn doctor-patient simulations. This approach offers a closer approximation to real clinical scenarios and allows for a detailed analysis of LLM behaviors in response to complex patient interactions. Our extensive experimental validation demonstrates the effectiveness of the AIE framework, with outcomes that align well with human evaluations, underscoring its potential to revolutionize medical LLM testing for improved healthcare delivery.
Synthetic or Authentic? Building Mental Patient Simulators from Longitudinal Evidence
SJTU Paris Elite Institute of Technology（上海交通大学巴黎精英理工学院）；X-LANCE Lab（X-LANCE实验室）；School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算机与信息系统学院）
Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance
Aalto University（阿尔托大学）；Tencent（腾讯）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Hong Kong Polytechnic University（香港理工大学）；Aarhus University（奥胡斯大学）；Technical University of Munich（慕尼黑工业大学）
Voice-Interactive Surgical Agent for Multimodal Patient Data Control
Image Guided Precision Cancer Surgery Institute, College of Medicine, Korea University（影像引导精准癌症手术研究所，韩国大学医学院）；Department of Radiology, Massachusetts General Hospital（放射科，麻省总医院）；Department of Thoracic and Cardiovascular Surgery, Korea University Guro Hospital, College of Medicine, Korea University（胸外科和心血管外科，韩国大学Guro医院，韩国大学医学院）；Department of Biomedical Sciences, College of Medicine, Korea University（生物医学科学系，韩国大学医学院）
Transforming Tuberculosis Care: Optimizing Large Language Models For Enhanced Clinician-Patient Communication
School of Engineering and Technology, University of Washington Tacoma（华盛顿大学塔科马分校工程与技术学院）；Institute for Clinical Effectiveness and Health Policy（临床效果与健康政策研究所）；Centre for Research on Epidemiology and Public Health (CIESP), CONICET（CONICET流行病学与公共卫生研究中心）；School of Nursing, University of Washington Seattle（华盛顿大学西雅图分校护理学院）；School of Nursing and Healthcare Leadership, University of Washington Tacoma（华盛顿大学塔科马分校护理与医疗保健领导学院）
PaniniQA: Enhancing Patient Education Through Interactive Question Answering
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Emory University（埃默里大学）；Northeastern University（东北大学）；UMass Chan Medical School（马萨诸塞大学陈医学院）；University of Minnesota（明尼苏达大学）；University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）
AmarDoctor: An AI-Driven, Multilingual, Voice-Interactive Digital Health Application for Primary Care Triage and Patient Management to Bridge the Digital Health Divide for Bengali Speakers
MedAi Bangladesh Limited（MedAi孟加拉有限公司）；Central Police Hospital, Dhaka, Bangladesh（达卡警察医院）；University of Birmingham, UK（英国伯明翰大学）
SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks
Tsinghua University（清华大学）；Chongqing University（重庆大学）；Peking University（北京大学）；ZenoMind AI；Xi’an Jiaotong University（西安交通大学）；Beijing Institute of Technology（北京理工大学）；Southeast University（东南大学）；Shanghai Jiao Tong University（上海交通大学）；Joy Future Academy（京东探索研究院）；The University of Hong Kong（香港大学）
