MCA：模态组合感知用于鲁棒的组合多模态检索
MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval
AI总结：
针对统一编码器在组合多模态检索中易受模态捷径影响的问题，提出模态组合感知框架，通过偏好损失和组合正则化增强鲁棒性，实验验证了其有效性。
摘要：
多模态检索旨在跨文本或图像等模态检索相关内容，支持从AI搜索到内容生成等应用。尽管像CLIP这样的分离编码器方法通过对比学习成功地对齐了模态特定的嵌入，但最近的多模态大语言模型（MLLMs）使得统一编码器能够直接处理组合输入。虽然这些方法灵活且先进，但我们发现，使用传统对比学习训练的统一编码器容易学习到模态捷径，导致在分布偏移下鲁棒性较差。我们提出了一种模态组合感知框架来缓解这一问题。具体而言，一个偏好损失强制多模态嵌入优于其单模态对应物，而一个组合正则化目标将多模态嵌入与从其单模态部分组合而成的原型对齐。这些目标显式地建模了组合表示与其单模态对应物之间的结构关系。在多个基准上的实验表明，在分布外检索中取得了提升，突显了模态组合感知作为利用MLLMs作为统一编码器时实现鲁棒组合多模态检索的有效原则。
英文摘要：
Multimodal retrieval, which seeks to retrieve relevant content across modalities such as text or image, supports applications from AI search to contents production. Despite the success of separate-encoder approaches like CLIP aligning modality-specific embeddings with contrastive learning, recent multimodal large language models (MLLMs) enable a unified encoder that directly processes composed inputs. While flexible and advanced, we identify that unified encoders trained with conventional contrastive learning are prone to learn modality shortcut, leading to poor robustness under distribution shifts. We propose a modality composition awareness framework to mitigate this issue. Concretely, it consists of a preference loss enforces multimodal embeddings to outperform their unimodal counterparts, and a composition regularization objective aligns multimodal embeddings with prototypes composed from its unimodal parts. These objectives explicitly model structural relationships between the composed representation and its unimodal counterparts. Experiments on various benchmarks show gains in out-of-distribution retrieval, highlighting modality composition awareness as a effective principle for robust composed multimodal retrieval when utilizing MLLMs as the unified encoder.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
