Large language models (LLMs) constitute a breakthrough state-of-the-art Artificial Intelligence technology which is rapidly evolving and promises to aid in medical diagnosis. However, the correctness and the accuracy of their returns has not yet been properly evaluated. In this work, we propose an LLM evaluation paradigm that incorporates two independent steps of a novel methodology, namely (1) multimodal LLM evaluation via structured interactions and (2) follow-up, domain-specific analysis based on data extracted via the previous interactions. Using this paradigm, (1) we evaluate the correctness and accuracy of LLM-generated medical diagnosis with publicly available multimodal multiple-choice questions(MCQs) in the domain of Pathology and (2) proceed to a systemic and comprehensive analysis of extracted results. We used GPT-4-Vision-Preview as the LLM to respond to complex, medical questions consisting of both images and text, and we explored a wide range of diseases, conditions, chemical compounds, and related entity types that are included in the vast knowledge domain of Pathology. GPT-4-Vision-Preview performed quite well, scoring approximately 84\% of correct diagnoses. Next, we further analyzed the findings of our work, following an analytical approach which included Image Metadata Analysis, Named Entity Recognition and Knowledge Graphs. Weaknesses of GPT-4-Vision-Preview were revealed on specific knowledge paths, leading to a further understanding of its shortcomings in specific areas. Our methodology and findings are not limited to the use of GPT-4-Vision-Preview, but a similar approach can be followed to evaluate the usefulness and accuracy of other LLMs and, thus, improve their use with further optimization.
Marking the Wrong Symptoms: Evaluating LLM Watermarks in Medical Texts
ETH Zurich（苏黎世联邦理工学院）；Berlin Institute of Health at Charité – Universitätsmedizin Berlin（柏林夏里特医学院柏林健康研究所）；Deutsches Herzzentrum der Charité – Medical Heart Center of Charité and German Heart Institute Berlin（柏林夏里特医学院德国心脏中心与柏林德国心脏研究所）
AnchorSIPS: A Synthetic Dataset and Evaluation Resource for Evidence-Supported Psychosis-Risk Symptom Measurement
Monash University（莫纳什大学）；Orygen（奥瑞金）；The University of Melbourne（墨尔本大学）；Swinburne University of Technology（斯威本科技大学）；University of Liverpool（利物浦大学）；Khalifa University（哈利法大学）
Identifying Symptoms of Delirium from Clinical Narratives Using Natural Language Processing
University of Florida（佛罗里达大学）；UF Health Shands Hospital（UF Health Shands医院）；University of California Los Angeles（加州大学洛杉矶分校）；UF Clinical and Translational Science Institute（UF临床与转化科学研究所）；North Florida/South Georgia Veterans Health Service（北佛罗里达/南乔治亚退伍军人卫生服务处）；College of Nursing, University of Florida（佛罗里达大学护理学院）；College of Public Health and Health Professions & College of Medicine, University of Florida（佛罗里达大学公共卫生与健康专业学院及医学院）
Enhancing Depression-Diagnosis-Oriented Chat with Psychological State Tracking
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；Shanghai Institute of Al for Education, East China Normal University（东华大学上海人工智能教育研究所）；School of Psychology and Cognitive Science, East China Normal University（东华大学心理学与认知科学学院）
Uncertainty-Aware Large Language Models for Explainable Disease Diagnosis
University of Minnesota（明尼苏达大学）；University of Chicago（芝加哥大学）；Columbia University（哥伦比亚大学）；University of Virginia（弗吉尼亚大学）；University of California San Francisco（加州大学旧金山分校）；Mayo Clinic（梅奥诊所）；Institute for Health Informatics（健康信息学研究所）
SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosis
College of Information and Intelligence, Hunan Agricultural University（湖南农业大学信息与智能学院）；College of Artificial Intelligence, Nankai University（南开大学人工智能学院）
A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosis
Zhejiang University-University of Illinois at Urbana-Champaign Institute（浙江大学伊利诺伊大学厄巴纳香槟校区联合学院）；Zhejiang University（浙江大学）；National University of Singapore（新加坡国立大学）；Angelalign Inc.（时代天使公司）
Fusing Medical Image Features and Clinical Features with Deep Learning for Computer-Aided Diagnosis
Beijing Institute of Technology（北京理工大学）；Xuanwu Hospital, Capital Medical University（首都医科大学宣武医院）；National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院肿瘤医院（国家癌症中心/国家肿瘤临床医学研究中心））
ISPO: An Integrated Ontology of Symptom Phenotypes for Semantic Integration of Traditional Chinese Medical Data
Beijing Jiaotong University（北京交通大学）；Hubei Provincial Hospital of Traditional Chinese Medicine（湖北省中医院）；China Academy of Chinese Medical Sciences（中国中医科学院）；Hubei University of Chinese Medicine（湖北中医药大学）；Guang'anmen Hospital, China Academy of Chinese Medical Sciences（中国中医科学院广安门医院）；Zhejiang University（浙江大学）
End-to-End Knowledge-Routed Relational Dialogue System for Automatic Diagnosis
Sun Yat-Sen University（中山大学）；Soochow University（苏州大学）；Dark Matter AI Inc.（暗物质人工智能公司）
除了目前越来越受到关注的会话聊天机器人或面向任务的对话系统之外，我们还在开发一种用于自动医疗诊断的对话系统，该系统与患者对话，收集他们自我报告之外的其他症状，并自动进行诊断。除了会话对话系统面临的挑战（如话题转换连贯性和问题理解）外，自动医疗诊断还对医学知识和疾病关系背景下的对话合理性提出了更严格的要求。现有的对话系统（Madotto，Wu，and Fung 2018; Wei et al. 2018; Li et al. 2017）主要依赖于数据驱动的学习，无法对额外的专家知识图谱进行编码。在这项工作中，我们提出了一个端到端的知识路由的关系对话系统（KR-DS），无缝地将丰富的医学知识图谱到对话管理中的主题转换，并使其与自然语言理解和自然语言生成合作。提出了一种新的知识路由深度Q网络（KR-DQN）来管理主题转换，它集成了一个用于编码不同症状和疾病对之间关系的关系精化分支和一个用于主题决策的知识路由图分支。在公共医疗对话数据集上进行的广泛实验表明，我们的KR-DS显著击败了最先进的方法（诊断准确率超过8%）。我们进一步展示了我们的KR-DS在新收集的医疗对话系统数据集上的优越性，该数据集在保留患者和医生之间的原始自我报告和对话数据方面更具挑战性。
