增强基础视觉语言模型对缺失模态的鲁棒性：可扩展的扩散用于双向特征恢复
Enhancing Foundation VLM Robustness to Missing Modality: Scalable Diffusion for Bi-directional Feature Restoration
AI总结：
本文提出一种通用缺失模态恢复策略，通过增强扩散模型实现双向特征对齐，提升视觉语言模型在模态缺失情况下的鲁棒性和可扩展性。
摘要：
视觉语言模型（VLM）通常假设在推理过程中具有完整的模态输入。然而，当某些模态不可用或不完整时，其效果会显著下降。当前缺失模态研究面临两大困境：基于提示的方法难以恢复缺失但关键的特征并降低VLM的泛化能力；基于填补的方法缺乏有效指导，容易生成语义无关的噪声。恢复精确语义的同时保持VLM的泛化能力仍具挑战性。因此，本文提出了一种通用的缺失模态恢复策略。我们引入增强的扩散模型作为可插拔的中阶段训练模块，以有效恢复缺失特征。我们的策略引入了两个关键创新：（I）动态模态门控，通过适应性利用条件特征指导生成语义一致的特征；（II）跨模态互学习机制，连接双模型的语义空间以实现双向对齐。值得注意的是，我们的策略保持了预训练VLM的原始完整性，无需对骨干模型进行微调，同时显著提高了对信息丢失的抗性。在基准数据集上的零样本评估表明，我们的方法在各种缺失率和条件下均优于现有基线，证明了其作为稳健且可扩展的扩展方法，确保了VLM在多样化缺失情况下的可靠性。我们的代码和模型将公开发布。
英文摘要：
Vision Language Model (VLM) typically assume complete modality input during inference. However, their effectiveness drops sharply when certain modalities are unavailable or incomplete. Current research on missing modality primarily faces two dilemmas: Prompt-based methods struggle to restore missing yet indispensable features and degrade the generalizability of VLM. Imputation-based approaches, lacking effective guidance, are prone to generating semantically irrelevant noise. Restoring precise semantics while sustaining VLM's generalization remains challenging. Therefore, we propose a general missing modality restoration strategy in this paper. We introduce an enhanced diffusion model as a pluggable mid-stage training module to effectively restore missing features. Our strategy introduces two key innovations: (I) Dynamic Modality Gating, which adaptively leverages conditional features to guide the generation of semantically consistent features; (II) Cross-Modal Mutual Learning mechanism, which bridges the semantic spaces of the dual models to achieve bi-directional alignment. Notably, our strategy maintains the original integrity of the pre-trained VLM, requiring no fine-tuning of the backbone models while significantly boosting resilience to information loss. Zero-shot evaluations across benchmark datasets demonstrate that our approach consistently outperforms existing baselines, establishing it as a robust and scalable extension that ensures VLM reliability across diverse missing rates and conditions. Our code and models will be publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
