先重建后查询：基于分解提示协作的持续缺失模态学习
Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaboration
AI总结：
针对持续微调中模态缺失导致性能下降和灾难性遗忘问题，提出RebQ框架，通过分解提示协作和利用预训练知识重建缺失模态，显著提升精度并降低遗忘。
摘要：
预训练的大规模多模态模型（LMMs）通过微调来适应多样化的用户应用。然而，由于传感器停用（例如，出于隐私或技术问题而关闭摄像头），微调可能面临挑战，导致模态不完整的数据，进而造成训练数据与推理数据之间的不一致。此外，持续训练会导致灾难性遗忘，稀释预训练LMMs中的知识。为克服这些挑战，我们引入了一个新任务——持续缺失模态学习（CMML），以研究在持续微调过程中某些模态数据缺失时模型如何泛化。我们的初步基准测试显示，现有方法在CMML中性能显著下降，即使借助先进的持续学习技术也是如此。因此，我们设计了一个名为“先重建后查询”（RebQ）的框架。它将提示分解为特定模态的提示，并将其拆分为存储在可通过键-查询机制访问的池中的组件，这有助于参数高效微调，并增强后续任务的知识可迁移性。同时，我们的RebQ利用预训练LMMs的广泛多模态知识来重建缺失模态的数据。综合实验表明，RebQ有效地重建了缺失模态信息并保留了预训练知识。具体而言，与基线相比，RebQ将平均精度从20.00提升至50.92，并将平均遗忘率从75.95降至8.56。代码和数据集可在https://github.com/Tree-Shu-Zhao/RebQ.pytorch获取。
英文摘要：
Pre-trained large multi-modal models (LMMs) exploit fine-tuning to adapt diverse user applications. Nevertheless, fine-tuning may face challenges due to deactivated sensors (e.g., cameras turned off for privacy or technical issues), yielding modality-incomplete data and leading to inconsistency in training data and the data for inference. Additionally, continuous training leads to catastrophic forgetting, diluting the knowledge in pre-trained LMMs. To overcome these challenges, we introduce a novel task, Continual Missing Modality Learning (CMML), to investigate how models can generalize when data of certain modalities is missing during continual fine-tuning. Our preliminary benchmarks reveal that existing methods suffer from a significant performance drop in CMML, even with the aid of advanced continual learning techniques. Therefore, we devise a framework termed Reconstruct before Query (RebQ). It decomposes prompts into modality-specific ones and breaks them into components stored in pools accessible via a key-query mechanism, which facilitates ParameterEfficient Fine-Tuning and enhances knowledge transferability for subsequent tasks. Meanwhile, our RebQ leverages extensive multi-modal knowledge from pre-trained LMMs to reconstruct the data of missing modality. Comprehensive experiments demonstrate that RebQ effectively reconstructs the missing modality information and retains pre-trained knowledge. Specifically, compared with the baseline, RebQ improves average precision from 20.00 to 50.92 and decreases average forgetting from 75.95 to 8.56. Code and datasets are available on https://github.com/Tree-Shu-Zhao/RebQ.pytorch
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
