弥合感知-认知鸿沟：通过希尔伯特-马amba重新工程SAM2以实现鲁棒的基于VLM的医学诊断
Bridging the Perception-Cognition Gap:Re-engineering SAM2 with Hilbert-Mamba for Robust VLM-based Medical Diagnosis
AI总结：
通过重新设计SAM2架构，引入希尔伯特空间填充曲线和希尔伯特-马amba交叉注意力机制，提升基于VLM的医学图像分割与诊断准确性。
摘要：
近期研究表明，视觉语言模型（VLMs）在自动化医学诊断等任务中具有巨大潜力。然而，处理复杂的三维（3D）多模态医学图像存在显著挑战——特别是互补信息的有效整合以及对微妙但关键的病理特征的偶尔忽视。为了解决这些问题，我们提出了一种新的两阶段融合框架，称为希尔伯特-VLM。该框架利用希尔伯特Med-SAM模块进行精确的病变分割，生成的多模态增强提示随后引导VLM进行准确的疾病分类。我们的关键创新在于对Segment Anything Model 2（SAM2）架构的系统性重新设计：我们将希尔伯特空间填充曲线纳入Mamba状态空间模型（SSM）的扫描机制中，以最大化3D数据中空间局部性的保留，这对于医学图像分析至关重要。我们还引入了新的希尔伯特-马amba交叉注意力（HMCA）机制和一个规模感知的解码器，以捕捉细粒度细节。同时，提示增强模块将分割掩码及其相应的文本属性统一为信息密集的提示，以支持VLM推理。进行了广泛的实验以验证希尔伯特-VLM模型的有效性。在BraTS2021分割基准测试中，它实现了82.35%的Dice分数，诊断分类准确率（ACC）为78.85%。这些结果表明，所提出的模型在提高基于VLM的医学分析的准确性和可靠性方面具有显著潜力。
英文摘要：
Recent studies suggest that Visual Language Models (VLMs) hold great potential for tasks such as automated medical diagnosis. However, processing complex three-dimensional (3D) multimodal medical images poses significant challenges - specifically, the effective integration of complementary information and the occasional oversight of subtle yet critical pathological features. To address these issues, we present a novel two-stage fusion framework termed Hilbert-VLM. This framework leverages the HilbertMed-SAM module for precise lesion segmentation, with the generated multimodal enhanced prompts then guiding the VLM toward accurate disease classification. Our key innovation lies in the systematic redesign of the Segment Anything Model 2 (SAM2) architecture: we incorporate Hilbert space-filling curves into the scanning mechanism of the Mamba State Space Model (SSM) to maximize the preservation of spatial locality in 3D data, a property critical for medical image analysis. We also introduce a novel Hilbert-Mamba Cross-Attention (HMCA) mechanism and a scale-aware decoder to capture fine-grained details. Meanwhile, the prompt enhancement module unifies segmentation masks and their corresponding textual attributes into an information-dense prompt to support VLM inference. Extensive experiments were conducted to validate the effectiveness of the Hilbert-VLM model. On the BraTS2021 segmentation benchmark, it achieves a Dice score of 82.35 percent, with a diagnostic classification accuracy (ACC) of 78.85 percent. These results demonstrate that the proposed model offers substantial potential to improve the accuracy and reliability of medical VLM-based analysis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
