缺失模态预测：通过单模态模型的联合嵌入实现非配对多模态学习
Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Models
AI总结：
针对多模态学习中模态缺失问题，提出融合单模态预训练模型参数高效微调与自监督联合嵌入的框架，通过提示微调预测缺失模态嵌入，在多个基准数据集上验证了有效性与鲁棒性。
摘要：
多模态学习通常依赖于训练和推理阶段所有模态都完全可用的假设。然而，在现实场景中，由于各种因素，持续获取完整的多模态数据面临重大挑战。这常常导致模态缺失问题，即某些模态的数据不存在，这不仅对多模态预训练模型的可用性构成巨大障碍，也对其微调以及在下游任务中保持鲁棒性构成巨大障碍。为了解决这些挑战，我们提出了一种新颖的框架，该框架将单模态预训练模型的参数高效微调与自监督联合嵌入学习方法相结合。该框架使模型能够在推理阶段预测表示空间中缺失模态的嵌入。我们的方法通过提示微调，利用可用模态的信息，有效地预测缺失的嵌入。我们在多个多模态基准数据集上评估了我们的方法，并证明了其在各种模态缺失场景下的有效性和鲁棒性。
英文摘要：
Multimodal learning typically relies on the assumption that all modalities are fully available during both the training and inference phases. However, in real-world scenarios, consistently acquiring complete multimodal data presents significant challenges due to various factors. This often leads to the issue of missing modalities, where data for certain modalities are absent, posing considerable obstacles not only for the availability of multimodal pretrained models but also for their fine-tuning and the preservation of robustness in downstream tasks. To address these challenges, we propose a novel framework integrating parameter-efficient fine-tuning of unimodal pretrained models with a self-supervised joint-embedding learning method. This framework enables the model to predict the embedding of a missing modality in the representation space during inference. Our method effectively predicts the missing embedding through prompt tuning, leveraging information from available modalities. We evaluate our approach on several multimodal benchmark datasets and demonstrate its effectiveness and robustness across various scenarios of missing modalities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
