VividMed：面向医学、具备通用视觉定位能力的视觉语言模型
VividMed: Vision Language Model with Versatile Visual Grounding for Medicine
AI总结：
本文提出医学视觉语言模型 VividMed，通过三阶段训练与自动数据合成支持 2D/3D 多模态语义分割、实例框定位，并提升 VQA 和报告生成性能。
摘要：
近期，视觉语言模型（VLM）的进展在生成具备视觉定位的回答方面展现出显著潜力。然而，其在医学领域的应用受到特有挑战的阻碍。例如，大多数 VLM 依赖单一的视觉定位方法，而复杂的医学任务需要更加通用的方法。此外，尽管大多数 VLM 只能处理 2D 图像，但很大一部分医学图像是 3D 的。医学数据的缺乏进一步加剧了这些障碍。为应对这些挑战，我们提出 VividMed，这是一种面向医学、具备通用视觉定位能力的视觉语言模型。我们的模型支持生成语义分割掩码和实例级边界框，并适配多种成像模态，包括 2D 和 3D 数据。我们设计了三阶段训练流程，以及基于开放数据集和模型的自动数据合成流水线。除视觉定位任务外，VividMed 在其他常见下游任务中也表现优异，包括视觉问答（VQA）和报告生成。消融实验从经验上表明，视觉定位能力的融合提升了这些任务上的性能。我们的代码已公开发布于 https://github.com/function2-llx/MMMM。
英文摘要：
Recent advancements in Vision Language Models (VLMs) have demonstrated remarkable promise in generating visually grounded responses. However, their application in the medical domain is hindered by unique challenges. For instance, most VLMs rely on a single method of visual grounding, whereas complex medical tasks demand more versatile approaches. Additionally, while most VLMs process only 2D images, a large portion of medical images are 3D. The lack of medical data further compounds these obstacles. To address these challenges, we present VividMed, a vision language model with versatile visual grounding for medicine. Our model supports generating both semantic segmentation masks and instance-level bounding boxes, and accommodates various imaging modalities, including both 2D and 3D data. We design a three-stage training procedure and an automatic data synthesis pipeline based on open datasets and models. Besides visual grounding tasks, VividMed also excels in other common downstream tasks, including Visual Question Answering (VQA) and report generation. Ablation studies empirically show that the integration of visual grounding ability leads to improved performance on these tasks. Our code is publicly available at https://github.com/function2-llx/MMMM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
