MicarVLMoE：一种用于医学图像字幕生成与报告生成的现代门控交叉对齐视觉语言混合专家模型
MicarVLMoE: A Modern Gated Cross-Aligned Vision-Language Mixture of Experts Model for Medical Image Captioning and Report Generation
AI总结：
提出视觉语言混合专家模型MicarVLMoE，通过多尺度视觉编码器、多头双分支潜在注意力模块和调制混合专家解码器，解决医学图像报告生成中的特征提取与多模态对齐问题，并在多种医学影像数据集上达到最先进性能。
摘要：
医学图像报告（MIR）旨在从放射图像生成结构化的临床描述。现有方法在细粒度特征提取、多模态对齐以及跨多种成像类型的泛化方面存在困难，通常依赖 vanilla transformers 且主要关注胸部 X 光片。我们提出了 MicarVLMoE，一种具有门控交叉对齐融合的视觉语言混合专家模型，旨在解决这些局限性。我们的架构包括：（i）一个多尺度视觉编码器（MSVE），用于在不同分辨率下捕捉解剖细节；（ii）一个多头双分支潜在注意力（MDLA）模块，通过潜在瓶颈表示实现视觉语言对齐；以及（iii）一个调制混合专家（MoE）解码器，用于自适应专家特化。我们将 MIR 扩展到 CT 扫描、视网膜成像、MRI 扫描和大体病理图像，在 COVCTR、MMR、PGROSS 和 ROCO 数据集上报告了最先进的结果。大量实验和消融研究证实了临床准确性、跨模态对齐和模型可解释性的提升。代码可在 https://github.com/AI-14/micar-vl-moe 获取。
英文摘要：
Medical image reporting (MIR) aims to generate structured clinical descriptions from radiological images. Existing methods struggle with fine-grained feature extraction, multimodal alignment, and generalization across diverse imaging types, often relying on vanilla transformers and focusing primarily on chest X-rays. We propose MicarVLMoE, a vision-language mixture-of-experts model with gated cross-aligned fusion, designed to address these limitations. Our architecture includes: (i) a multiscale vision encoder (MSVE) for capturing anatomical details at varying resolutions, (ii) a multihead dual-branch latent attention (MDLA) module for vision-language alignment through latent bottleneck representations, and (iii) a modulated mixture-of-experts (MoE) decoder for adaptive expert specialization. We extend MIR to CT scans, retinal imaging, MRI scans, and gross pathology images, reporting state-of-the-art results on COVCTR, MMR, PGROSS, and ROCO datasets. Extensive experiments and ablations confirm improved clinical accuracy, cross-modal alignment, and model interpretability. Code is available at https://github.com/AI-14/micar-vl-moe.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
