Sim4Seg：利用区域感知视觉-语言相似性掩码提升多模态多疾病医学诊断分割
Sim4Seg: Boosting Multimodal Multi-disease Medical Diagnosis Segmentation with Region-Aware Vision-Language Similarity Masks
- Lingran Song, Yucheng Zhou, Jianbing Shen（作者）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本文提出医学诊断分割任务及M3DS数据集，并设计Sim4Seg框架，利用区域感知视觉-语言相似性掩码和测试时扩展策略，同时提升多模态多疾病医学图像的分割与诊断性能。
AI中文摘要：
尽管像素级医学图像分析取得了显著进展，但现有医学图像分割模型很少联合探索医学分割与诊断任务。然而，对患者而言，模型能够同时提供可解释的诊断和医学分割结果至关重要。本文提出了一项医学视觉-语言任务，称为医学诊断分割（Medical Diagnosis Segmentation, MDS），旨在理解针对医学图像的临床查询，并生成相应的分割掩码及诊断结果。为促进该任务，我们首先构建了多模态多疾病医学诊断分割（Multimodal Multi-disease Medical Diagnosis Segmentation, M3DS）数据集，该数据集包含多样的多模态多疾病医学图像，并配对相应的分割掩码和诊断思维链，通过自动化的诊断思维链生成流程创建。此外，我们提出了Sim4Seg，这是一个新颖的框架，通过利用区域感知视觉-语言相似性到掩码（Region-Aware Vision-Language Similarity to Mask, RVLS2M）模块来提升诊断分割性能。为改善整体表现，我们研究了针对MDS任务的测试时扩展策略。实验结果表明，我们的方法在分割和诊断两方面均优于基线方法。
英文摘要：
Despite significant progress in pixel-level medical image analysis, existing medical image segmentation models rarely explore medical segmentation and diagnosis tasks jointly. However, it is crucial for patients that models can provide explainable diagnoses along with medical segmentation results. In this paper, we introduce a medical vision-language task named Medical Diagnosis Segmentation (MDS), which aims to understand clinical queries for medical images and generate the corresponding segmentation masks as well as diagnostic results. To facilitate this task, we first present the Multimodal Multi-disease Medical Diagnosis Segmentation (M3DS) dataset, containing diverse multimodal multi-disease medical images paired with their corresponding segmentation masks and diagnosis chain-of-thought, created via an automated diagnosis chain-of-thought generation pipeline. Moreover, we propose Sim4Seg, a novel framework that improves the performance of diagnosis segmentation by taking advantage of the Region-Aware Vision-Language Similarity to Mask (RVLS2M) module. To improve overall performance, we investigate a test-time scaling strategy for MDS tasks. Experimental results demonstrate that our method outperforms the baselines in both segmentation and diagnosis.
