AI 中文总结
针对多模态肿瘤生存预测中模态缺失与整合难题，提出图引导的专家混合框架，利用图结构管理多模型集成，在TCGA-LUAD数据集上超越单模态及普通集成方法。
AI 中文摘要
大型语言模型（LLMs）在需要少量演示示例的任务中展现出令人印象深刻的能力，使其成为有效的少样本学习者。尽管具有潜力，但LLMs在处理涉及多种模态或推理步骤的复杂现实世界任务时面临挑战。例如，基于临床数据、细胞切片和基因组学预测癌症患者的生存期，带来了显著的逻辑复杂性。尽管已提出多种方法应对这些挑战，但它们往往因无法同时考虑所有模态，或无法处理模态缺失、模态变化以及多模态数据整合问题，而难以取得理想性能，最终削弱了其有效性。本论文提出了一种新颖的多模态肿瘤生存预测方法以解决这些局限性。受近期LLMs进展的启发，特别是基于专家混合（MoE）的模型，引入了一个图引导的MoE框架。该框架利用图结构有效管理预测，并组合多个模型以增强预测能力。该方法并非训练单一基础模型进行端到端生存预测，而是利用MoE引导的集成来自动管理模型调用作为工具。通过利用现有模型的优势并在MoE框架下引导它们，目标是在复杂现实世界任务中实现更好的性能和更准确的预测。在TCGA-LUAD数据集上的实验和分析表明，该方法优于单个模态模型和普通集成模型。
英文摘要
Large Language Models (LLMs) have displayed impressive capabilities in handling tasks that require few demonstration examples, making them effective few-shot learners. Despite their potential, LLMs face challenges when it comes to addressing complex real-world tasks that involve multiple modalities or reasoning steps. For example, predicting cancer patients' survival period based on clinical data, cell slides, and genomics poses significant logistical complexities. Although several approaches have been proposed to tackle these challenges, they often fall short in achieving promising performance due to their inability to consider all modalities simultaneously or account for missing modalities, variations in modalities, and the integration of multi-modal data, ultimately compromising their effectiveness. This thesis proposes a novel approach for multi-modal tumor survival prediction to address these limitations. Taking inspiration from recent advancements in LLMs, particularly Mixture of Experts (MoE)-based models, a graph-guided MoE framework is introduced. This framework utilizes a graph structure to manage the predictions effectively and combines multiple models to enhance predictive power. Rather than training a single foundation model for end-to-end survival prediction, the approach leverages a MOE-guided ensemble to manage model callings as tools automatically. By leveraging the strengths of existing models and guiding them through a MOE framework, the aim is to achieve better performance and more accurate predictions in complex real-world tasks. Experiments and analysis on the TCGA-LUAD dataset show improved performance over the individual modal and vanilla ensemble models.
