AliFuse：对齐与融合多模态医疗数据以用于计算机辅助诊断
AliFuse: Aligning and Fusing Multi-modal Medical Data for Computer-Aided Diagnosis
AI总结：
本文提出基于Transformer的AliFuse框架，通过注意力机制与结合恢复建模和对比学习的方法对齐并融合多模态医疗数据，在阿尔茨海默病分类任务中取得优于八个基线的最优性能。
摘要：
为诊断决策收集的医疗数据通常是多模态的，可提供关于受试者的全面信息。尽管计算机辅助诊断系统能从多模态输入中获益，但有效融合此类数据仍是一项具有挑战性的任务，也是医学研究的关键焦点。本文提出了一种基于Transformer的框架，称为AliFuse，用于对齐和融合多模态医疗数据。具体而言，我们将医学图像以及非结构化和结构化临床记录转换为视觉和语言token，采用模态内和模态间注意力机制来学习所有成像与非成像数据的统一表示以进行分类。此外，我们将恢复建模与对比学习框架相集成，联合学习图像与文本间的高层语义对齐，以及借助另一模态实现对某一模态的低层理解。我们将AliFuse应用于阿尔茨海默病分类，在五个公开数据集上取得了最先进的性能，并优于八个基线方法。
英文摘要：
Medical data collected for diagnostic decisions are typically multimodal, providing comprehensive information on a subject. While computer-aided diagnosis systems can benefit from multimodal inputs, effectively fusing such data remains a challenging task and a key focus in medical research. In this paper, we propose a transformer-based framework, called Alifuse, for aligning and fusing multimodal medical data. Specifically, we convert medical images and both unstructured and structured clinical records into vision and language tokens, employing intramodal and intermodal attention mechanisms to learn unified representations of all imaging and non-imaging data for classification. Additionally, we integrate restoration modeling with contrastive learning frameworks, jointly learning the high-level semantic alignment between images and texts and the low-level understanding of one modality with the help of another. We apply Alifuse to classify Alzheimer's disease, achieving state-of-the-art performance on five public datasets and outperforming eight baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
