arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.15134cs.CVcs.AIcs.LG

超越标量距离:来自冻结MLLM的语义属性梯度用于视觉嵌入

Beyond Scalar Distances: Semantic Attribute Gradients from Frozen MLLMs for Visual Embeddings

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Shubhang Bhatnagar, Dheeraj Baiju, Narendra Ahuja

AI总结:

提出SAGA框架,利用冻结的多模态大语言模型(MLLM)通过GRPO奖励机制为视觉编码器提供属性级监督,替代传统标量距离,提升零样本图像检索性能。

摘要:

用于检索的视觉编码器通常通过类标签监督进行训练:每个训练对简化为一个标量,均匀地将嵌入推远或拉近,就好像每个视觉属性要么不同要么匹配。一个多模态大语言模型(MLLM),在展示相同的一对图像时,能够阐述这些属性并利用它们预测图像是否共享一个类别。我们提出\textbf{SAGA},一个框架,将这种基于语言、属性感知的感知转化为编码器本身的训练信号。具体来说,我们使用组相对策略优化(GRPO)来奖励MLLM对视觉编码器令牌的正确预测。由于正确的预测要求这些令牌暴露该对之间不同或匹配的具体属性,梯度推动编码器编码这些属性,用属性解析的监督取代统一的成对标量。一个辅助的注意力蒸馏损失将编码器的嵌入锚定到MLLM关注的令牌上,一个标准的度量学习损失塑造嵌入几何结构以进行最近邻检索。MLLM在整个过程中被冻结,在推理时被丢弃,与度量学习基线的部署成本相匹配。在CUB-200-2011、Cars-196、FGVC-Aircraft和iNaturalist Aves上的零样本图像检索中,SAGA在Recall@1上比最先进的基线提高了3到6个百分点。

英文摘要:

Vision encoders for retrieval are typically trained with class-label supervision: each training pair reduces to a scalar that uniformly pushes the embedding apart or pulls it together, as if every visual attribute either differed or matched. A multimodal large language model (MLLM), shown the same pair, can articulate those attributes and use them to predict whether the images share a class. We propose \textbf{SAGA}, a framework that turns this language-grounded, attribute-aware perception into a training signal for the encoder itself. Specifically, we use Group Relative Policy Optimization (GRPO) to reward the MLLM for correct predictions on the vision encoder's tokens. Since correct predictions require those tokens to expose the specific attributes that differ or match between the pair, the gradient pushes the encoder to encode them, replacing the uniform pair-level scalar with attribute-resolved supervision. An auxiliary attention-distillation loss anchors the encoder's embedding to tokens the MLLM attended to, and a standard metric-learning loss shapes the embedding geometry for nearest-neighbour retrieval. The MLLM is frozen throughout and discarded at inference, matching the deployment cost of a metric-learning baseline. SAGA improves Recall@1 by 3 to 6 points over state-of-the-art baselines on CUB-200-2011, Cars-196, FGVC-Aircraft, and iNaturalist Aves on zero-shot image retrieval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑