自适应全局与细粒度感知融合用于兼容硬负样本放大的人脸嵌入
Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplification
AI总结：
本文提出AGFF-Embed方法，通过自适应融合全局和细粒度语义信息，提升多模态嵌入在一般和细粒度理解上的性能。
摘要：
多模态嵌入在对齐视觉和语言方面起着桥梁作用，两种主要实现——基于CLIP和基于MLLM的嵌入模型——都局限于只能捕捉全局语义信息。尽管众多研究聚焦于细粒度理解，但我们发现当前MLLM嵌入所针对的复杂场景往往包含全局和细粒度元素的混合感知模式，因此需要一种兼容的融合机制。在本文中，我们提出了一种自适应全局与细粒度感知融合用于MLLM嵌入的方法（AGFF-Embed），该方法促使MLLM生成多个聚焦于不同语义维度的嵌入，然后进行自适应和流畅的聚合。此外，我们还将AGFF-Embed与显式梯度放大（EGA）技术相结合，以在批次内实现硬负样本增强，而无需对数据集进行细粒度编辑。在MMEB和MMVP-VLM基准测试中，AGFF-Embed在一般和细粒度理解方面均实现了优于其他多模态嵌入模型的最先进性能。
英文摘要：
Multimodal embeddings serve as a bridge for aligning vision and language, with the two primary implementations -- CLIP-based and MLLM-based embedding models -- both limited to capturing only global semantic information. Although numerous studies have focused on fine-grained understanding, we observe that complex scenarios currently targeted by MLLM embeddings often involve a hybrid perceptual pattern of both global and fine-grained elements, thus necessitating a compatible fusion mechanism. In this paper, we propose Adaptive Global and Fine-grained perceptual Fusion for MLLM Embeddings (AGFF-Embed), a method that prompts the MLLM to generate multiple embeddings focusing on different dimensions of semantic information, which are then adaptively and smoothly aggregated. Furthermore, we adapt AGFF-Embed with the Explicit Gradient Amplification (EGA) technique to achieve in-batch hard negatives enhancement without requiring fine-grained editing of the dataset. Evaluation on the MMEB and MMVP-VLM benchmarks shows that AGFF-Embed comprehensively achieves state-of-the-art performance in both general and fine-grained understanding compared to other multimodal embedding models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
