面向多模态学习的知识驱动子空间融合与梯度协调方法
Knowledge-driven Subspace Fusion and Gradient Coordination for Multi-modal Learning
AI总结：
针对多模态学习在癌症诊断中难以有效建模基因组学与组织学数据关联的问题，提出知识驱动子空间融合与梯度协调框架，在胶质瘤三项下游任务中性能超越现有最优方法。
摘要：
多模态学习在癌症诊断与预后中发挥着关键作用。当前基于深度学习的多模态方法往往受限于其对基因组学与组织学数据间复杂关联的建模能力，难以应对肿瘤生态系统中肿瘤与微环境共同促成恶性表型的内在复杂性。我们提出一种具备生物学可解释性且鲁棒的多模态学习框架，通过分解组织学图像与基因组学的特征子空间（分别对应不同的肿瘤特征与微环境特征），实现两类数据的高效融合。为增强跨模态交互，我们设计了知识驱动的子空间融合方案，包含跨模态可变形注意力模块与基因引导的一致性策略。此外，为实现子空间知识的动态优化，我们进一步提出一种新型梯度协调学习策略。大量实验验证了所提方法的有效性，在胶质瘤诊断、肿瘤分级和生存分析三项下游任务中均优于现有最优技术。我们的代码已发布于https://github.com/helenypzhang/Subspace-Multimodal-Learning。
英文摘要：
Multi-modal learning plays a crucial role in cancer diagnosis and prognosis. Current deep learning based multi-modal approaches are often limited by their abilities to model the complex correlations between genomics and histology data, addressing the intrinsic complexity of tumour ecosystem where both tumour and microenvironment contribute to malignancy. We propose a biologically interpretative and robust multi-modal learning framework to efficiently integrate histology images and genomics by decomposing the feature subspace of histology images and genomics, reflecting distinct tumour and microenvironment features. To enhance cross-modal interactions, we design a knowledge-driven subspace fusion scheme, consisting of a cross-modal deformable attention module and a gene-guided consistency strategy. Additionally, in pursuit of dynamically optimizing the subspace knowledge, we further propose a novel gradient coordination learning strategy. Extensive experiments demonstrate the effectiveness of the proposed method, outperforming state-of-the-art techniques in three downstream tasks of glioma diagnosis, tumour grading, and survival analysis. Our code is available at https://github.com/helenypzhang/Subspace-Multimodal-Learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
