用于图像-组学分类的基因诱导多模态预训练
Gene-induced Multimodal Pre-training for Image-omic Classification
AI总结：
本文提出基因诱导多模态预训练框架，通过分组多头自注意力编码器和掩码块建模分别处理基因与全切片图像，并利用三元组学习实现高阶特征融合，在TCGA数据集上图像-组学分类准确率达99.47%。
摘要：
结合基因组学检测的肿瘤微环境组织学分析是现代医学中大多数癌症的金标准。本文提出了一种基因诱导多模态预训练框架，该框架联合结合基因组学和全切片图像用于分类任务。本研究旨在应对多模态图像-组学分类的主要挑战，即（1）从千兆像素WSI和数万个基因中进行患者级特征提取的困难，以及（2）考虑高阶相关性建模的有效融合。具体而言，我们首先提出一种分组多头自注意力基因编码器，以捕获基因表达队列中的全局结构化特征。我们设计了掩码块建模范式来捕获不同组织的潜在病理特征。该掩码策略是随机掩码WSI的块嵌入的固定长度连续子序列。最后，我们结合配对模态的分类标记，并提出一个三元组学习模块来学习高阶相关性和判别性患者级信息。预训练后，可采用简单的微调来获得分类结果。在TCGA数据集上的实验结果表明了我们的网络架构和预训练框架的优越性，在图像-组学分类中达到99.47%的准确率。代码公开于https://github.com/huangwudiduan/GIMP。
英文摘要：
Histology analysis of the tumor micro-environment integrated with genomic assays is the gold standard for most cancers in modern medicine. This paper proposes a Gene-induced Multimodal Pre-training (GiMP) framework, which jointly incorporates genomics and Whole Slide Images (WSIs) for classification tasks. Our work aims at dealing with the main challenges of multi-modality image-omic classification w.r.t. (1) the patient-level feature extraction difficulties from gigapixel WSIs and tens of thousands of genes, and (2) effective fusion considering high-order relevance modeling. Concretely, we first propose a group multi-head self-attention gene encoder to capture global structured features in gene expression cohorts. We design a masked patch modeling paradigm (MPM) to capture the latent pathological characteristics of different tissues. The mask strategy is randomly masking a fixed-length contiguous subsequence of patch embeddings of a WSI. Finally, we combine the classification tokens of paired modalities and propose a triplet learning module to learn high-order relevance and discriminative patient-level information.After pre-training, a simple fine-tuning can be adopted to obtain the classification results. Experimental results on the TCGA dataset show the superiority of our network architectures and our pre-training framework, achieving 99.47% in accuracy for image-omic classification. The code is publicly available at https://github.com/huangwudiduan/GIMP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
