利用混合几何注意力推进异构医学数据上的多模态融合
Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention
浏览论文内容
中文总结 AI 辅助
针对异构医学数据多模态融合难题，提出CURE框架，通过HyFuse层及相关模块逐步整合模态，有效捕捉跨模态交互，降低计算成本，在多数据集上评估显示其性能显著提升，优于领先方法。
中文摘要 AI 辅助
多模态融合学习（MFL）在医学领域潜力巨大，但现有策略面临挑战。它们难以有效捕捉复杂跨模态交互，计算成本高，且针对特定模态配置设计评估。本文提出CURE框架，通过HyFuse层逐步整合模态，该层含高效残差卷积模块和混合空间注意力混合器。还采用互补模块学习稳健共享表示。在16个公共数据集上评估表明，CURE性能提升3.97%，计算成本降低87.8%。
英文摘要
Multimodal fusion learning (MFL) has shown great potential in the medical domain, where we are faced with disparate data modalities such as imaging, clinical records, and omics. However, existing MFL strategies face several major challenges. First, they struggle to capture complex cross-modal interactions effectively, which in turn limits performance improvements. Second, they incur high computational costs, restricting their applicability in resource-constrained healthcare AI applications. Finally, they are often designed and evaluated for narrow, fixed modality configurations (e.g., imaging-only, or specific pairs such as image and omics), which limits evidence of their adaptability and generalizability to broader collections of heterogeneous medical modalities. To address these challenges, we propose a novel MFL framework - Cascaded Unified Representation Learning for Efficient Fusion Network (CURE) - a lightweight and scalable framework that progressively integrates various modalities through a novel efficient Hybrid Geometry Aware Fusion layer (HyFuse), where each HyFuse layer is sequentially learned for each modality, making the framework adaptable and generalizable. Within HyFuse, an efficient residual convolution module captures rich multi-scale features to ensure cost-effective learning, while a hybrid-space aware attention mixer learns coarse-to-fine structural cues to better preserve cross-modal relationships. Complementary learnable late-fusion and shared information refinement modules are then employed to learn robust modality-order-invariant shared representations, which in turn yields consistent performance improvements. Extensive evaluations on 16 public datasets show that CURE outperforms leading multimodal fusion methods, boosting performance by up to 3.97% and lowering computational costs by up to 87.8%, ensuring more effective and reliable predictions.
发表机构
- Indian Institute of Technology Ropar（印度理工学院罗帕尔分校）
- RoentGen Health（伦琴健康公司）
- Deakin University（迪肯大学）
- University of Central Florida（中佛罗里达大学）
- Queensland University of Technology（昆士兰科技大学）
- Murdoch University（莫道克大学）
机构由 AI 辅助整理，请以论文原文为准。
