arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.15183cs.LGcs.CL
收藏

看见以泛化:视觉数据如何纠正绑定捷径

Seeing to Generalize: How Visual Data Corrects Binding Shortcuts

Department of Computer Science, Pontificia Universidad Católica, Santiago, Chile(计算机科学系,天主教大学,圣地亚哥,智利)

Nicolas Buzeta, Felipe del Rio, Cristian Hinostroza, Denis Parra, Hans Lobel, Rodrigo Toro Icarte

AI总结:

视觉数据训练可增强模型在单模态任务中的推理与泛化能力,通过改变绑定策略提升分布外性能。

摘要:

视觉语言模型(VLMs)是通过扩展大语言模型(LLMs)来增加视觉能力的,但在本工作中我们观察到一个令人惊讶的现象:VLMs在纯文本任务上可以超越其底层LLMs,特别是在长上下文信息检索方面。为了研究这种效果,我们构建了一个受控的合成检索任务,并发现一个仅在文本上训练的变压器模型在分布内准确率达到完美,但在分布外泛化时却失败,而随后在图像令牌化的相同任务上训练,几乎使文本-only的分布外性能翻倍。机制可解释性揭示了视觉训练改变了模型的内部绑定策略:纯文本训练鼓励位置捷径,而基于图像的训练则通过空间翻译不变性破坏这些捷径,迫使模型采用更稳健的符号绑定机制,即使在重新引入纯文本示例后也能保持。我们进一步描述了在不同训练制度、视觉编码器和初始设置下绑定策略的变化,并展示了在预训练LLM到VLM过渡期间也出现类似的转变。我们的发现表明,跨模态训练可以增强推理和泛化能力,即使对于单模态任务也是如此。

英文摘要:

Vision Language Models (VLMs) are designed to extend Large Language Models (LLMs) with visual capabilities, yet in this work we observe a surprising phenomenon: VLMs can outperform their underlying LLMs on purely text-only tasks, particularly in long-context information retrieval. To investigate this effect, we build a controlled synthetic retrieval task and find that a transformer trained only on text achieves perfect in-distribution accuracy but fails to generalize out of distribution, while subsequent training on an image-tokenized version of the same task nearly doubles text-only OOD performance. Mechanistic interpretability reveals that visual training changes the model's internal binding strategy: text-only training encourages positional shortcuts, whereas image-based training disrupts them through spatial translation invariance, forcing the model to adopt a more robust symbolic binding mechanism that persists even after text-only examples are reintroduced. We further characterize how binding strategies vary across training regimes, visual encoders, and initializations, and show that analogous shifts occur during pretrained LLM-to-VLM transitions. Our findings suggest that cross-modal training can enhance reasoning and generalization even for tasks grounded in a single modality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑