arXiv 2410.150742024-10-19cs.CVcs.AI

LLaVA-Ultra：面向超声的大型中文语言与视觉助手

LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound

ZJU-UIUC Institute, Zhejiang University（浙江大学ZJU-UIUC学院）；University of Washington（华盛顿大学）；Zhejiang University School of Medicine Sir Run Run Shaw Hospital（浙江大学医学院附属邵逸夫医院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）

Xuechen Guo, Wenhao Chai, Shi-Yan Li, Gaoang Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出LLaVA-Ultra，通过细粒度视觉融合和知识蒸馏筛选冗余医学图像，并结合大规模中文超声指令数据微调，在多个Med-VQA数据集上超越现有最优模型。

摘要：

多模态大语言模型（MLLM）近来已成为备受关注的研究焦点。通过利用强大的大语言模型（LLM），它推动了对话式生成人工智能从单模态文本向执行多模态任务的转变。这一热潮开始对医学领域产生显著影响。然而，通用视觉语言模型（VLM）对医学视觉问答（Med-VQA）缺乏精细的理解能力。即使是专门为医学领域定制的模型，也倾向于产生含糊且视觉相关性较弱的回答。本文提出了一种通过参数高效微调实现中文医学视觉对话的细粒度自适应VLM架构。具体而言，我们设计了一个带有细粒度视觉编码器的融合模块，以增强对细微医学视觉语义的理解。随后，我们注意到大多数先前工作忽略了医学场景中常见的数据冗余问题。在单个文本对应多张图像的情况下，我们利用加权评分与知识蒸馏来自适应地筛选出与文本描述相匹配的有效图像。在实现方面，我们利用从医院获取的大规模多模态中文超声数据集。我们基于专业医生撰写的文本构建指令跟随数据，从而确保有效的微调。凭借增强的模型和高质量数据，我们的大型中文语言与视觉超声助手（LLaVA-Ultra）展现出对医学场景的强大能力和鲁棒性。在三个Med-VQA数据集上，LLaVA-Ultra在多项指标上超越了先前最先进的模型。

英文摘要：

Multimodal Large Language Model (MLLM) has recently garnered attention as a prominent research focus. By harnessing powerful LLM, it facilitates a transition of conversational generative AI from unimodal text to performing multimodal tasks. This boom begins to significantly impact medical field. However, general visual language model (VLM) lacks sophisticated comprehension for medical visual question answering (Med-VQA). Even models specifically tailored for medical domain tend to produce vague answers with weak visual relevance. In this paper, we propose a fine-grained adaptive VLM architecture for Chinese medical visual conversations through parameter-efficient tuning. Specifically, we devise a fusion module with fine-grained vision encoders to achieve enhancement for subtle medical visual semantics. Then we note data redundancy common to medical scenes is ignored in most prior works. In cases of a single text paired with multiple figures, we utilize weighted scoring with knowledge distillation to adaptively screen valid images mirroring text descriptions. For execution, we leverage a large-scale multimodal Chinese ultrasound dataset obtained from the hospital. We create instruction-following data based on text from professional doctors, which ensures effective tuning. With enhanced model and quality data, our Large Chinese Language and Vision Assistant for Ultrasound (LLaVA-Ultra) shows strong capability and robustness to medical scenarios. On three Med-VQA datasets, LLaVA-Ultra surpasses previous state-of-the-art models on various metrics.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。