arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.04170cs.CV
收藏

M2EF-NNs:用于癌症生存预测的多模态多实例证据融合神经网络

M2EF-NNs: Multimodal Multi-instance Evidence Fusion Neural Networks for Cancer Survival Prediction

City University of Macau(澳门城市大学);Guangxi Medical University(广西医科大学);Nantong University(南通大学)

Hui Luo, Jiashuang Huang, Hengrong Ju, Tianyi Zhou, Weiping Ding

AI总结:

本文提出M2EF-NNs模型,通过ViT提取病理图像全局特征,利用多模态注意力融合基因组与图像信息,并首次引入Dempster-Shafer证据理论估计模态不确定性,在TCGA数据集上显著提升了癌症生存预测的准确性与可靠性。

摘要:

准确的癌症生存预测对于辅助临床医生制定治疗方案至关重要。包括组织病理学图像和基因组数据在内的多模态数据提供了互补且全面的信息,能够极大提升该任务的准确性。然而,现有方法尽管取得了有前景的结果,但仍存在两个显著局限:它们未能有效利用全局上下文,并且忽视了模态不确定性。在本研究中,我们提出了一种名为M2EF-NNs的神经网络模型,该模型利用多模态和多实例证据融合技术实现准确的癌症生存预测。具体而言,为了捕获图像中的全局信息,我们使用预训练的Vision Transformer (ViT)模型获取组织病理学图像的图像块特征嵌入。然后,我们引入一个多模态注意力模块,该模块以基因组嵌入作为查询,学习基因组与组织病理学图像之间的协同注意力映射,从而实现多模态信息的早期交互融合,并更好地捕获它们之间的相关性。随后,我们首次将Dempster-Shafer证据理论(DST)应用于癌症生存预测。我们利用处理后的多模态特征对类别概率分布进行参数化,并引入主观逻辑来估计不同模态相关的不确定性。通过与Dempster-Shafer理论相结合,我们能够动态调整多模态融合后类别概率的权重,以实现可信的生存预测。最后,在TCGA数据集上的实验验证证实了我们所提出的方法在癌症生存预测方面取得的显著改进,并增强了模型的可靠性。

英文摘要:

Accurate cancer survival prediction is crucial for assisting clinical doctors in formulating treatment plans. Multimodal data, including histopathological images and genomic data, offer complementary and comprehensive information that can greatly enhance the accuracy of this task. However, the current methods, despite yielding promising results, suffer from two notable limitations: they do not effectively utilize global context and disregard modal uncertainty. In this study, we put forward a neural network model called M2EF-NNs, which leverages multimodal and multi-instance evidence fusion techniques for accurate cancer survival prediction. Specifically, to capture global information in the images, we use a pre-trained Vision Transformer (ViT) model to obtain patch feature embeddings of histopathological images. Then, we introduce a multimodal attention module that uses genomic embeddings as queries and learns the co-attention mapping between genomic and histopathological images to achieve an early interaction fusion of multimodal information and better capture their correlations. Subsequently, we are the first to apply the Dempster-Shafer evidence theory (DST) to cancer survival prediction. We parameterize the distribution of class probabilities using the processed multimodal features and introduce subjective logic to estimate the uncertainty associated with different modalities. By combining with the Dempster-Shafer theory, we can dynamically adjust the weights of class probabilities after multimodal fusion to achieve trusted survival prediction. Finally, Experimental validation on the TCGA datasets confirms the significant improvements achieved by our proposed method in cancer survival prediction and enhances the reliability of the model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑