多模态大语言模型增强的跨语言跨模态检索
Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval
AI总结：
本文提出LECCR方法，利用多模态大语言模型生成多视图语义槽以增强视觉特征，并结合英语引导的软化匹配，有效提升了跨语言跨模态检索中视觉与非英语表示的对齐效果。
摘要：
跨语言跨模态检索（CCR）旨在基于非英语查询检索视觉相关内容，且无需在训练期间依赖人工标注的跨模态数据对。一种流行的方法是利用机器翻译（MT）创建伪平行数据对，从而建立视觉与非英语文本数据之间的对应关系。然而，由于视觉与文本之间存在显著的语义鸿沟，以及预训练编码器和数据噪声导致非英语表示质量较低，对齐它们的表示面临诸多挑战。为克服这些挑战，我们提出LECCR，这是一种新颖的解决方案，引入多模态大语言模型（MLLM）来改善视觉与非英语表示之间的对齐。具体而言，我们首先利用MLLM生成详细的视觉内容描述，并将其聚合为多视图语义槽，以封装不同语义。然后，我们将这些语义槽作为内部特征，并利用它们与视觉特征进行交互。通过这种方式，我们增强了视觉特征中的语义信息，缩小了模态间的语义鸿沟，并生成局部视觉语义用于后续的多层次匹配。此外，为进一步增强视觉与非英语特征之间的对齐，我们引入了英语引导下的软化匹配。该方法在视觉与非英语特征之间提供了更全面、更可靠的跨模态对应关系。在四个CCR基准（即Multi30K、MSCOCO、VATEX和MSR-VTT-CN）上的大量实验证明了我们所提方法的有效性。代码：\url{https://github.com/LiJiaBei-7/leccr}。
英文摘要：
Cross-lingual cross-modal retrieval (CCR) aims to retrieve visually relevant content based on non-English queries, without relying on human-labeled cross-modal data pairs during training. One popular approach involves utilizing machine translation (MT) to create pseudo-parallel data pairs, establishing correspondence between visual and non-English textual data. However, aligning their representations poses challenges due to the significant semantic gap between vision and text, as well as the lower quality of non-English representations caused by pre-trained encoders and data noise. To overcome these challenges, we propose LECCR, a novel solution that incorporates the multi-modal large language model (MLLM) to improve the alignment between visual and non-English representations. Specifically, we first employ MLLM to generate detailed visual content descriptions and aggregate them into multi-view semantic slots that encapsulate different semantics. Then, we take these semantic slots as internal features and leverage them to interact with the visual features. By doing so, we enhance the semantic information within the visual features, narrowing the semantic gap between modalities and generating local visual semantics for subsequent multi-level matching. Additionally, to further enhance the alignment between visual and non-English features, we introduce softened matching under English guidance. This approach provides more comprehensive and reliable inter-modal correspondences between visual and non-English features. Extensive experiments on four CCR benchmarks, \ie Multi30K, MSCOCO, VATEX, and MSR-VTT-CN, demonstrate the effectiveness of our proposed method. Code: \url{https://github.com/LiJiaBei-7/leccr}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
