局部对齐改进视觉-语言模型
Locality Alignment Improves Vision-Language Models
AI总结：
针对视觉语言模型（VLMs）因预训练视觉主干网络缺乏局部语义而导致空间推理错误的问题，提出高效后训练阶段局部对齐及微调方法MaskEmbed，通过掩码重建损失学习图像块语义贡献，显著提升模型在空间理解等基准上的性能。
摘要：
视觉语言模型（VLMs）近年来得到了越来越广泛的应用，但许多模型在基本的空间推理上仍然存在错误。我们假设这是由于VLMs采用了预训练的视觉主干网络，具体来说，是使用图像级监督和最小归纳偏置训练的视觉变换器。这类模型可能无法编码图像中每个位置的类别内容，我们的目标是通过一个能有效捕获局部和全局图像语义的视觉主干网络来解决这一问题。我们的主要洞察是，我们不需要新的监督来学习这种能力——预训练模型包含了丰富的局部语义知识，我们可以提取并利用这些知识进行可扩展的自监督。我们为ViTs提出了一种新的高效后训练阶段，称为局部对齐，以及一种名为MaskEmbed的新型微调过程，该过程使用掩码重建损失来学习每个图像块的语义贡献。我们首先使用纯视觉基准对局部对齐进行评估，发现它提升了模型在块级语义分割上的性能，特别是对于使用图像-标题对训练的强主干网络（例如CLIP和SigLIP）。然后，我们训练了一系列带有和不带有局部对齐的VLMs，并表明局部对齐的主干网络在一系列基准上提升了性能，尤其是那些涉及空间理解的基准（例如RefCOCO、OCID-Ref、TallyQA、VSR、AI2D）。总体而言，我们证明了可以通过局部对齐阶段高效地学习局部语义提取，并且这一过程有益于使用现成视觉主干网络的VLM训练方案。
英文摘要：
Vision language models (VLMs) have seen growing adoption in recent years, but many still struggle with basic spatial reasoning errors. We hypothesize that this is due to VLMs adopting pre-trained vision backbones, specifically vision transformers (ViTs) trained with image-level supervision and minimal inductive biases. Such models may fail to encode the class contents at each position in the image, and our goal is to resolve this with a vision backbone that effectively captures both local and global image semantics. Our main insight is that we do not require new supervision to learn this capability - pre-trained models contain significant knowledge of local semantics that we can extract and use for scalable self-supervision. We propose a new efficient post-training stage for ViTs called locality alignment and a novel fine-tuning procedure called MaskEmbed that uses a masked reconstruction loss to learn semantic contributions for each image patch. We first evaluate locality alignment with a vision-only benchmark, finding that it improves a model's performance at patch-level semantic segmentation, especially for strong backbones trained with image-caption pairs (e.g., CLIP and SigLIP). We then train a series of VLMs with and without locality alignment, and show that locality-aligned backbones improve performance across a range of benchmarks, particularly ones that involve spatial understanding (e.g., RefCOCO, OCID-Ref, TallyQA, VSR, AI2D). Overall, we demonstrate that we can efficiently learn local semantic extraction via a locality alignment stage, and that this procedure benefits VLM training recipes that use off-the-shelf vision backbones.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
