arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.17823cs.CVcs.AIcs.LG
收藏

鲁棒多模态学习:基于跨模态代理令牌

Robust Multimodal Learning via Cross-Modal Proxy Tokens

University of California Riverside(加州大学河滨分校);Amazon(亚马逊)

Md Kaykobad Reza, Ameya Patil, Mashhour Solh, M. Salman Asif

AI总结:

针对多模态推理时模态缺失导致性能下降的问题,提出跨模态代理令牌方法,利用低秩适配器与对齐损失近似缺失模态类别令牌,在五个数据集上超越现有基线,兼顾鲁棒性与效率。

摘要:

多模态模型在推理过程中当一个或多个模态缺失时,性能通常会显著下降。为应对这一挑战,我们提出了一种简单而有效的方法,在保持所有模态可用时强性能的同时,增强对缺失模态的鲁棒性。我们的方法引入了跨模态代理令牌(CMPTs),通过仅关注可用模态的令牌来近似缺失模态的类别令牌,无需显式的模态生成或辅助网络。为了以最小的计算开销高效学习这些近似,我们在冻结的单模态编码器中采用低秩适配器,并联合优化对齐损失与任务特定损失。在五个多模态数据集上的大量实验表明,我们的方法在各种缺失率下优于最先进的基线,同时在完整模态设置中取得了具有竞争力的结果。总体而言,我们的方法为鲁棒多模态学习提供了一种灵活且高效的解决方案。本文代码可在 https://github.com/CSIPlab/Cross-Modal-Proxy-Tokens 获取。

英文摘要:

Multimodal models often experience a significant performance drop when one or more modalities are missing during inference. To address this challenge, we propose a simple yet effective approach that enhances robustness to missing modalities while maintaining strong performance when all modalities are available. Our method introduces cross-modal proxy tokens (CMPTs), which approximate the class token of a missing modality by attending only to the tokens of the available modality without requiring explicit modality generation or auxiliary networks. To efficiently learn these approximations with minimal computational overhead, we employ low-rank adapters in frozen unimodal encoders and jointly optimize an alignment loss with a task-specific loss. Extensive experiments on five multimodal datasets show that our method outperforms state-of-the-art baselines across various missing rates while achieving competitive results in complete-modality settings. Overall, our method offers a flexible and efficient solution for robust multimodal learning. The code for this paper is available at: https://github.com/CSIPlab/Cross-Modal-Proxy-Tokens.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑