持续跨模态泛化
Continual Cross-Modal Generalization
AI总结：
本文提出持续跨模态泛化方法，通过CMoE-Adapter和伪模态重放机制，利用中介模态将新模态增量映射到共享离散码本，在保留先验知识的同时实现强跨模态泛化性能。
摘要：
跨模态泛化旨在从多模态对中学习共享的离散表示空间，从而实现跨未标注模态的知识迁移。然而，为所有模态对实现统一表示需要大量配对数据，这通常不切实际。受丰富双模态数据（例如在 ImageBind 中）可用性的启发，我们探索了一种持续学习方法，通过中介模态将新模态增量映射到共享离散码本中。我们提出了持续混合专家适配器（CMoE-Adapter），将不同模态投影到统一空间，同时保留先验知识。为了跨阶段对齐语义，我们引入了伪模态重放（PMR）机制，结合动态扩展的码本，使模型能够利用已学习的模态作为指导，自适应地整合新模态。在图像-文本、音频-文本、视频-文本和语音-文本上的大量实验表明，我们的方法在各种跨模态泛化任务中取得了强劲的性能。代码在补充材料中提供。
英文摘要：
Cross-modal generalization aims to learn a shared discrete representation space from multimodal pairs, enabling knowledge transfer across unannotated modalities. However, achieving a unified representation for all modality pairs requires extensive paired data, which is often impractical. Inspired by the availability of abundant bimodal data (e.g., in ImageBind), we explore a continual learning approach that incrementally maps new modalities into a shared discrete codebook via a mediator modality. We propose the Continual Mixture of Experts Adapter (CMoE-Adapter) to project diverse modalities into a unified space while preserving prior knowledge. To align semantics across stages, we introduce a Pseudo-Modality Replay (PMR) mechanism with a dynamically expanding codebook, enabling the model to adaptively incorporate new modalities using learned ones as guidance. Extensive experiments on image-text, audio-text, video-text, and speech-text show that our method achieves strong performance on various cross-modal generalization tasks. Code is provided in the supplementary material.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
