BiPVL-Seg：用于医学图像分割的具有全局-局部对齐的双向渐进视觉-语言融合
BiPVL-Seg: Bidirectional Progressive Vision-Language Fusion with Global-Local Alignment for Medical Image Segmentation
AI总结：
提出BiPVL-Seg端到端框架，通过双向渐进融合和全局-局部对比对齐强化视觉与文本特征交互，在CT和MR多类医学图像分割任务上超越现有方法。
摘要：
医学图像分割通常仅依赖视觉数据，忽略了临床医生在诊断中使用的丰富文本信息。视觉语言模型试图弥合这一差距，但现有方法往往独立处理视觉和文本特征，导致跨模态对齐较弱。由于空间视觉特征与序列文本嵌入之间的固有差异，简单的融合技术会失败。此外，医学术语偏离通用语言，限制了现成文本编码器的有效性，并进一步阻碍了视觉-语言对齐。我们提出了BiPVL-Seg，这是一个端到端框架，通过架构和训练创新整合了视觉-语言融合和嵌入对齐，其中两个组件相互增强以提升医学图像分割。BiPVL-Seg在架构中引入了双向渐进融合，促进了视觉和文本编码器之间的分阶段信息交换。此外，它还结合了全局-局部对比对齐，这是一种训练目标，通过在类别和概念层面同时对齐文本和视觉嵌入来增强文本编码器的理解能力。在跨CT和MR模态的多样化医学成像基准上进行的大量实验表明，与最先进的方法相比，BiPVL-Seg在复杂的多类分割中表现出卓越的性能。源代码可在该GitHub存储库中获取。
英文摘要：
Medical image segmentation typically relies solely on visual data, overlooking the rich textual information clinicians use for diagnosis. Vision-language models attempt to bridge this gap, but existing approaches often process visual and textual features independently, resulting in weak cross-modal alignment. Simple fusion techniques fail due to the inherent differences between spatial visual features and sequential text embeddings. Additionally, medical terminology deviates from general language, limiting the effectiveness of off-the-shelf text encoders and further hindering vision-language alignment. We propose BiPVL-Seg, an end-to-end framework that integrates vision-language fusion and embedding alignment through architectural and training innovations, where both components reinforce each other to enhance medical image segmentation. BiPVL-Seg introduces bidirectional progressive fusion in the architecture, which facilitates stage-wise information exchange between vision and text encoders. Additionally, it incorporates global-local contrastive alignment, a training objective that enhances the text encoder's comprehension by aligning text and vision embeddings at both class and concept levels. Extensive experiments on diverse medical imaging benchmarks across CT and MR modalities demonstrate BiPVL-Seg's superior performance when compared with state-of-the-art methods in complex multi-class segmentation. Source code is available in this GitHub repository.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
