重新审视双编码视觉-语言模型中的组合性：推断的作用
Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference
AI总结：
本文探讨了双编码视觉-语言模型中组合性问题的根源，提出通过改进推断协议提升组合性能，引入轻量级变压器学习局部对齐，优于全微调和端到端训练方法。
摘要：
双编码视觉-语言模型（VLMs）如CLIP常被描述为词袋系统，因其在组合性基准上表现不佳。本文认为这一限制可能源于标准推断协议基于全局余弦相似度，而非表示不足。通过受控诊断实验，我们发现显式强制细粒度区域段对齐可显著提升组合性能而不更新预训练编码器。我们引入轻量级变压器直接从冻结的补丁和令牌嵌入中学习此类对齐。与全微调和先前端到端组合训练方法相比，尽管这些方法在域内检索中有所提升，但其增益在分布偏移下不一致。相比之下，学习冻结表示上的局部对齐在域内检索中匹配全微调，同时在受控域外组合性基准上带来显著改进。这些结果将全局嵌入匹配识别为双编码VLMs的关键瓶颈，并强调对齐机制对稳健组合泛化的重要性。
英文摘要：
Dual-encoder Vision-Language Models (VLMs) such as CLIP are often characterized as bag-of-words systems due to their poor performance on compositional benchmarks. We argue that this limitation may stem less from deficient representations than from the standard inference protocol based on global cosine similarity. First, through controlled diagnostic experiments, we show that explicitly enforcing fine-grained region-segment alignment at inference dramatically improves compositional performance without updating pretrained encoders. We then introduce a lightweight transformer that learns such alignments directly from frozen patch and token embeddings. Comparing against full fine-tuning and prior end-to-end compositional training methods, we find that although these approaches improve in-domain retrieval, their gains do not consistently transfer under distribution shift. In contrast, learning localized alignment over frozen representations matches full fine-tuning on in-domain retrieval while yielding substantial improvements on controlled out-of-domain compositional benchmarks. These results identify global embedding matching as a key bottleneck in dual-encoder VLMs and highlight the importance of alignment mechanisms for robust compositional generalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
