arXiv 2503.039872025-03-06cs.CVcs.AIcs.CLcs.LG

RetinalGPT：一个由大型视觉语言模型驱动的视网膜临床偏好对话助手

RetinalGPT: A Retinal Clinical Preference Conversational Assistant Powered by Large Vision-Language Models

School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）；School of Computing, Clemson University（克莱姆森大学计算机学院）；Washington University School of Medicine in St. Louis（圣路易斯华盛顿大学医学院）；The Keck School of Medicine of the University of Southern California（南加州大学凯克医学院）；Mayo Clinic（梅奥诊所）；Banner Alzheimer’s Institute（班纳阿尔茨海默病研究所）

Wenhui Zhu, Xin Li, Xiwen Chen, Peijie Qiu, Vamsi Krishna Vasa, Xuanzhao Dong, Yanxi Chen, Natasha Lepore, Oana Dumitrascu, Yi Su, Yalin Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出由大型视觉语言模型驱动的视网膜临床偏好对话助手RetinalGPT，通过构建大型数据集、新型数据管线与定制视觉指令微调，在8个基准数据集上大幅提升视网膜疾病诊断、定量分析与病灶定位性能，构建了端到端临床研究框架。

摘要：

近期，多模态大型语言模型（MLLMs）因其在处理和分析图像、视频和音频等非文本数据方面的卓越能力而备受关注。值得注意的是，已有研究探索了通用领域MLLMs向医疗领域的多种适配，如LLaVA-Med。然而，这些医疗适配模型在理解和解释视网膜图像方面仍不够先进。相比之下，医学专家强调定量分析对于疾病检测和解释的重要性。这凸显了通用领域与医疗领域MLLMs之间的差距：通用领域MLLMs在广泛应用中表现出色，但缺乏医疗领域精确诊断和解释任务所需的专门知识。为了应对这些挑战，我们引入了RetinalGPT，这是一个用于视网膜图像临床偏好定量分析的多模态对话助手。具体而言，我们通过汇编大型视网膜图像数据集、开发新型数据管线，并采用定制的视觉指令微调，以增强视网膜分析并丰富医学知识。特别地，RetinalGPT在8个基准视网膜数据集的视网膜疾病诊断中，大幅优于通用领域的MLLM。除了疾病诊断外，RetinalGPT还具备定量分析和病灶定位功能，代表了利用LLMs构建可解释的端到端临床研究框架的先驱性一步。代码可在https://github.com/Retinal-Research/RetinalGPT获取。

英文摘要：

Recently, Multimodal Large Language Models (MLLMs) have gained significant attention for their remarkable ability to process and analyze non-textual data, such as images, videos, and audio. Notably, several adaptations of general-domain MLLMs to the medical field have been explored, including LLaVA-Med. However, these medical adaptations remain insufficiently advanced in understanding and interpreting retinal images. In contrast, medical experts emphasize the importance of quantitative analyses for disease detection and interpretation. This underscores a gap between general-domain and medical-domain MLLMs: while general-domain MLLMs excel in broad applications, they lack the specialized knowledge necessary for precise diagnostic and interpretative tasks in the medical field. To address these challenges, we introduce \textit{RetinalGPT}, a multimodal conversational assistant for clinically preferred quantitative analysis of retinal images. Specifically, we achieve this by compiling a large retinal image dataset, developing a novel data pipeline, and employing customized visual instruction tuning to enhance both retinal analysis and enrich medical knowledge. In particular, RetinalGPT outperforms MLLM in the generic domain by a large margin in the diagnosis of retinal diseases in 8 benchmark retinal datasets. Beyond disease diagnosis, RetinalGPT features quantitative analyses and lesion localization, representing a pioneering step in leveraging LLMs for an interpretable and end-to-end clinical research framework. The code is available at https://github.com/Retinal-Research/RetinalGPT

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。