面向缺失模态持续适应的高效提示
Efficient Prompting for Continual Adaptation to Missing Modalities
AI总结：
将动态缺失模态问题表述为持续学习任务，提出模态特定、任务感知和任务特定三种提示及对比任务交互策略，在三个公开数据集上持续优于现有方法。
摘要：
缺失模态问题在现实应用中很常见，由设备故障和隐私担忧等因素引起。在具有缺失模态的下游数据集上微调预训练模型时，性能可能会显著下降。当前方法通常聚合各种缺失情况来训练恢复模块或对齐多模态特征，导致性能次优、计算成本高，以及在数据按顺序到达的持续环境中存在灾难性遗忘的风险。在本文中，我们将动态缺失模态问题表述为持续学习任务，并引入持续多模态缺失模态任务。为高效应对这一挑战，我们引入了三种提示：模态特定、任务感知和任务特定提示。这些提示使模型能够学习模态内、模态间、任务内和任务间特征。此外，我们提出了一种对比任务交互策略，以显式学习关联不同模态的提示。我们在三个公开数据集上进行了广泛实验，该方法始终优于最先进的方法。
英文摘要：
Missing modality issues are common in real-world applications, arising from factors such as equipment failures and privacy concerns. When fine-tuning pre-trained models on downstream datasets with missing modalities, performance can degrade significantly. Current methods often aggregate various missing cases to train recovery modules or align multimodal features, resulting in suboptimal performance, high computational costs, and the risk of catastrophic forgetting in continual environments where data arrives sequentially. In this paper, we formulate the dynamic missing modality problem as a continual learning task and introduce the continual multimodal missing modality task. To address this challenge efficiently, we introduce three types of prompts: modality-specific, task-aware, and task-specific prompts. These prompts enable the model to learn intra-modality, inter-modality, intra-task, and inter-task features. Furthermore, we propose a contrastive task interaction strategy to explicitly learn prompts correlating different modalities. We conduct extensive experiments on three public datasets, where our method consistently outperforms state-of-the-art approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
