基于核的无监督嵌入对齐以增强视觉-语言模型中的视觉表示
Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models
AI总结：
本文提出一种基于核的无监督对齐方法，将CLIP视觉表示与DINOv2对齐，在保持文本兼容性的同时增强细粒度感知，并提升零样本识别、空间推理及下游MLLM性能。
摘要：
视觉-语言模型（如CLIP）在对齐视觉与文本表示方面取得了显著成功，已成为许多多模态大语言模型（MLLM）（如LLaVA和OpenFlamingo）的重要组成部分。然而，大量研究指出CLIP有限的细粒度感知能力是一个关键缺陷，导致下游MLLM出现严重失败。相比之下，以视觉为中心的基础模型（如DINOv2）在捕捉图像细节方面展现出卓越能力。在本工作中，我们提出一种新颖的基于核的方法，将CLIP的视觉表示与DINOv2的视觉表示对齐，确保所得嵌入在保持与文本嵌入兼容的同时增强感知能力。我们的对齐目标被设计用于高效随机优化。经过这种仅图像的对齐微调后，视觉编码器保持与冻结文本编码器的兼容性，并在零样本物体识别、细粒度空间推理和定位方面表现出显著提升。通过集成对齐后的视觉编码器，下游MLLM也展现出增强的性能。
英文摘要：
Vision-language models, such as CLIP, have achieved significant success in aligning visual and textual representations, becoming essential components of many multi-modal large language models (MLLMs) like LLaVA and OpenFlamingo. However, numerous studies have identified CLIP's limited fine-grained perception as a critical drawback, leading to substantial failures in downstream MLLMs. In contrast, vision-centric foundation models like DINOv2 demonstrate remarkable capabilities in capturing fine details from images. In this work, we propose a novel kernel-based method to align CLIP's visual representation with that of DINOv2, ensuring that the resulting embeddings maintain compatibility with text embeddings while enhancing perceptual capabilities. Our alignment objective is designed for efficient stochastic optimization. Following this image-only alignment fine-tuning, the visual encoder retains compatibility with the frozen text encoder and exhibits significant improvements in zero-shot object recognition, fine-grained spatial reasoning, and localization. By integrating the aligned visual encoder, downstream MLLMs also demonstrate enhanced performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
