通过显式难负样本梯度放大改进多模态嵌入学习
Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying
AI总结：
本文通过分析info-NCE损失梯度，提出显式放大难负样本梯度的多模态嵌入学习方法，基于LLaVA-OneVision-7B在MMEB上取得最优性能，并与自研QQMM集成后登顶排行榜。
摘要：
随着近年来多模态大语言模型（MLLMs）的快速发展，基础性的对比语言-图像预训练（CLIP）框架已被成功扩展到MLLMs，从而为广泛的检索任务提供更强大、更通用的多模态嵌入。尽管取得了这些进展，从CLIP风格模型到MLLMs，核心对比学习范式基本保持不变。在该框架内，难负样本的有效挖掘仍然是提升性能的关键因素。先前工作引入了离线和在线难负样本挖掘策略，以提高对比学习效率。虽然这些方法带来了多模态嵌入的改进，但每个难负样本对学习过程的具体贡献尚未被深入研究。在本工作中，我们对info-NCE损失关于查询、正样本和负样本的梯度进行了详细分析，阐明了难负样本在更新模型参数中的作用。基于这一分析，我们提出显式放大与难负样本相关的梯度，从而促使模型学习更具判别力的嵌入。我们的多模态嵌入模型采用所提出的显式梯度放大器，并基于LLaVA-OneVision-7B架构训练，在MMEB基准上相比使用相同MLLM骨干网络的先前方法取得了最先进性能。此外，当与我们自研的MLLM QQMM集成时，我们的方法在MMEB排行榜上获得了第一名。代码和模型已在https://github.com/QQ-MM/QQMM-embed上发布。
英文摘要：
With the rapid advancement of multi-modal large language models (MLLMs) in recent years, the foundational Contrastive Language-Image Pretraining (CLIP) framework has been successfully extended to MLLMs, enabling more powerful and universal multi-modal embeddings for a wide range of retrieval tasks. Despite these developments, the core contrastive learning paradigm remains largely unchanged from CLIP-style models to MLLMs. Within this framework, the effective mining of hard negative samples continues to be a critical factor for enhancing performance. Prior works have introduced both offline and online strategies for hard negative mining to improve the efficiency of contrastive learning. While these approaches have led to improved multi-modal embeddings, the specific contribution of each hard negative sample to the learning process has not been thoroughly investigated. In this work, we conduct a detailed analysis of the gradients of the info-NCE loss with respect to the query, positive, and negative samples, elucidating the role of hard negatives in updating model parameters. Building upon this analysis, we propose to explicitly amplify the gradients associated with hard negative samples, thereby encouraging the model to learn more discriminative embeddings. Our multi-modal embedding model, trained with the proposed Explicit Gradient Amplifier and based on the LLaVA-OneVision-7B architecture, achieves state-of-the-art performance on the MMEB benchmark compared to previous methods utilizing the same MLLM backbone. Furthermore, when integrated with our self-developed MLLM, QQMM, our approach attains the top rank on the MMEB leaderboard. Code and models are released on https://github.com/QQ-MM/QQMM-embed.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
