面向不完整多模态生存预测的蒸馏提示学习
Distilled Prompt Learning for Incomplete Multimodal Survival Prediction
AI总结：
针对多模态生存预测中模态缺失的问题，提出两阶段蒸馏提示学习框架DisPro，结合LLMs鲁棒性补全缺失模态的共有与特有知识，在多种缺失场景下表现更优。
摘要：
病理图像、基因谱等多模态数据的融合被广泛应用于精准生存预测。尽管多模态生存模型近年来取得了进展，但收集用于多模态融合的完整模态数据仍是一大难题，阻碍了其在临床场景中的应用。现有处理不完整模态的方法往往存在不足，它们通常仅能补全缺失模态的有限部分知识。为解决这一问题，我们提出蒸馏提示学习框架（Distilled Prompt Learning，DisPro），利用大语言模型（Large Language Models, LLMs）对缺失模态的强鲁棒性，采用两阶段提示来补全缺失模态的全面信息。第一阶段为单模态提示（Unimodal Prompting，UniPro），蒸馏每个模态的知识分布，为后续阶段补充缺失模态的模态特有知识做准备。第二阶段为多模态提示（Multimodal Prompting，MultiPro），利用可用模态作为LLMs的提示来推断缺失模态，提供模态共有信息；同时将第一阶段获取的单模态知识注入多模态推断，补全缺失模态的模态特有知识。覆盖多种缺失场景的大量实验证明了所提方法的优越性。代码可在https://github.com/Innse/DisPro 获取。
英文摘要：
The integration of multimodal data including pathology images and gene profiles is widely applied in precise survival prediction. Despite recent advances in multimodal survival models, collecting complete modalities for multimodal fusion still poses a significant challenge, hindering their application in clinical settings. Current approaches tackling incomplete modalities often fall short, as they typically compensate for only a limited part of the knowledge of missing modalities. To address this issue, we propose a Distilled Prompt Learning framework (DisPro) to utilize the strong robustness of Large Language Models (LLMs) to missing modalities, which employs two-stage prompting for compensation of comprehensive information for missing modalities. In the first stage, Unimodal Prompting (UniPro) distills the knowledge distribution of each modality, preparing for supplementing modality-specific knowledge of the missing modality in the subsequent stage. In the second stage, Multimodal Prompting (MultiPro) leverages available modalities as prompts for LLMs to infer the missing modality, which provides modality-common information. Simultaneously, the unimodal knowledge acquired in the first stage is injected into multimodal inference to compensate for the modality-specific knowledge of the missing modality. Extensive experiments covering various missing scenarios demonstrated the superiority of the proposed method. The code is available at https://github.com/Innse/DisPro.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
