arXiv 2505.058042025-05-25cs.CV

描述医学图像中的任何内容

Describe Anything in Medical Images

Tulane University（杜兰大学）；Carnegie Mellon University（卡内基梅隆大学）；Northeastern University（东北大学）；Harvard Medical School（哈佛医学院）；University of Georgia（佐治亚大学）

Xi Xiao, Yunbei Zhang, Thanh-Huy Nguyen, Ba-Thinh Lam, Janet Wang, Lin Zhao, Jihun Hamm, Tianyang Wang, Xingjian Li, Xiao Wang, Hao Xu, Tianming Liu, Min Xu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对医学图像区域描述生成缺乏专门框架的问题，提出MedDAM，利用大型视觉-语言模型和专家提示词，在三个数据集上超越多个领先模型，验证了区域级语义对齐的关键作用。

摘要：

局部图像描述生成技术已取得显著进展，例如描述任何内容模型（DAM）能够在没有显式区域-文本监督的情况下生成详细的区域特定描述。然而，此类能力尚未广泛应用于医学成像等专业领域，在这些领域中，诊断解读依赖于细微的区域发现而非全局理解。为弥补这一差距，我们提出了MedDAM，这是首个利用大型视觉-语言模型进行医学图像区域特定描述生成的综合框架。MedDAM采用由医学专家设计的、针对特定成像模态定制的提示词，并建立了一个稳健的评估基准，该基准包含定制化评估协议、数据预处理流程和专用问答模板库。该基准评估MedDAM及其他可适配的大型视觉-语言模型，通过属性级验证任务聚焦于临床事实准确性，从而规避了医学数据集中缺乏真实区域-描述配对的问题。在VinDr-CXR、LIDC-IDRI和SkinCon数据集上的大量实验表明，MedDAM在该任务中优于领先的同类模型（包括GPT-4o、Claude 3.7 Sonnet、LLaMA-3.2 Vision、Qwen2.5-VL、GPT-4Rol和OMG-LLaVA），揭示了区域级语义对齐在医学图像理解中的重要性，并将MedDAM确立为临床视觉-语言整合的有前景基础。

英文摘要：

Localized image captioning has made significant progress with models like the Describe Anything Model (DAM), which can generate detailed region-specific descriptions without explicit region-text supervision. However, such capabilities have yet to be widely applied to specialized domains like medical imaging, where diagnostic interpretation relies on subtle regional findings rather than global understanding. To mitigate this gap, we propose MedDAM, the first comprehensive framework leveraging large vision-language models for region-specific captioning in medical images. MedDAM employs medical expert-designed prompts tailored to specific imaging modalities and establishes a robust evaluation benchmark comprising a customized assessment protocol, data pre-processing pipeline, and specialized QA template library. This benchmark evaluates both MedDAM and other adaptable large vision-language models, focusing on clinical factuality through attribute-level verification tasks, thereby circumventing the absence of ground-truth region-caption pairs in medical datasets. Extensive experiments on the VinDr-CXR, LIDC-IDRI, and SkinCon datasets demonstrate MedDAM's superiority over leading peers (including GPT-4o, Claude 3.7 Sonnet, LLaMA-3.2 Vision, Qwen2.5-VL, GPT-4Rol, and OMG-LLaVA) in the task, revealing the importance of region-level semantic alignment in medical image understanding and establishing MedDAM as a promising foundation for clinical vision-language integration.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。