UniMoCo：面向鲁棒多模态嵌入的统一模态补全
UniMoCo: Unified Modality Completion for Robust Multi-Modal Embeddings
AI总结：
本文提出 UniMoCo 视觉-语言架构，通过从文本生成视觉特征的模态补全模块及一致性训练策略，统一对齐多样查询-目标模态组合，提升多模态嵌入的鲁棒性并缓解训练数据偏差。
摘要：
当前研究已探索将视觉-语言模型用于多模态嵌入任务，例如信息检索、视觉定位和分类。然而，真实场景通常涉及查询与目标之间多样的模态组合，例如文本和图像到文本、文本和图像到文本和图像，以及文本到文本和图像。这些多样组合给现有模型带来了重大挑战，因为它们在训练期间难以在统一嵌入空间中对齐所有模态组合，从而降低推理时的性能。为解决这一局限，我们提出 UniMoCo，这是一种为多模态嵌入任务设计的新型视觉-语言模型架构。UniMoCo 引入了一个模态补全模块，可从文本输入生成视觉特征，确保查询和目标双方的模态完整性。此外，我们开发了专门的训练策略，以对齐来自原始输入和模态补全输入的嵌入，确保嵌入空间内的一致性。这使模型能够在各类嵌入任务中鲁棒地处理广泛的模态组合。实验表明，UniMoCo 优于以往方法，并在不同设置下表现出一致的鲁棒性。更重要的是，我们识别并量化了传统方法中由训练数据模态组合不平衡导致的固有偏差，该偏差可通过我们的模态补全范式得到缓解。代码可在 https://github.com/HobbitQia/UniMoCo 获取。
英文摘要：
Current research has explored vision-language models for multi-modal embedding tasks, such as information retrieval, visual grounding, and classification. However, real-world scenarios often involve diverse modality combinations between queries and targets, such as text and image to text, text and image to text and image, and text to text and image. These diverse combinations pose significant challenges for existing models, as they struggle to align all modality combinations within a unified embedding space during training, which degrades performance at inference. To address this limitation, we propose UniMoCo, a novel vision-language model architecture designed for multi-modal embedding tasks. UniMoCo introduces a modality-completion module that generates visual features from textual inputs, ensuring modality completeness for both queries and targets. Additionally, we develop a specialized training strategy to align embeddings from both original and modality-completed inputs, ensuring consistency within the embedding space. This enables the model to robustly handle a wide range of modality combinations across embedding tasks. Experiments show that UniMoCo outperforms previous methods while demonstrating consistent robustness across diverse settings. More importantly, we identify and quantify the inherent bias in conventional approaches caused by imbalance of modality combinations in training data, which can be mitigated through our modality-completion paradigm. The code is available at https://github.com/HobbitQia/UniMoCo.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
