桥接视觉语言模型中的隐藏状态
Bridging Hidden States in Vision-Language Models
AI总结：
针对现有视觉语言模型融合方式的局限，本文提出轻量跨模态双向注意力融合模块BRIDGE，直接对齐双模态隐藏状态，在多类基准上性能更优且保留双编码器效率。
摘要：
视觉语言模型（VLMs）是一类将图像内容与自然语言对齐的新型模型。现有方法通常采用两类融合方式：一是早期融合，即在编码器内部混合token或特征；二是晚期融合，即比较池化后的嵌入。许多方法还将融合过程与自回归解码器绑定。然而，两种模态的隐藏状态本身已承载丰富的模态专属结构（视觉的空间布局、文本的句法与语义），因此直接对齐这些状态是匹配两种模态“认知内容”的自然路径。本文提出一种轻量融合模块：在两个编码器靠近顶部的位置设置若干仅跨模态的双向注意力层。每一层将视觉和文本编码器的隐藏状态序列投影到共享空间，执行跨模态注意力，再通过门控残差更新回传，同时搭配简单的稳定器提升对齐效果。编码器保持非因果特性，具备强理解能力，生成过程则通过可选解码器实现干净解耦。在标准检索、视觉问答（VQA）和视觉推理基准测试中，BRIDGE模型性能优于同级别VLMs，同时保留了对比模型的双编码器效率。本文代码已公开于https://github.com/jfeinashley/BRIDGE。
英文摘要：
Vision-Language Models (VLMs) are a new family of models that align image content with natural language. Existing approaches typically fuse either (a) early: by mixing tokens/features inside the encoders, or (b) late: by comparing pooled embeddings. Many methods also tie fusion to an autoregressive decoder. However, the hidden states of both modalities already carry rich, modality-specific structure (spatial layout in vision; syntax and semantics in text), so directly aligning these states is a natural way to match what the two modalities "think". We propose a lightweight fusion module: a few cross-only, bidirectional attention layers placed near the top of both encoders. Each layer projects the vision and text encoder hidden-state sequences into a shared space, attends across modalities, and sends gated residual updates back, with simple stabilizers to improve alignment. The encoders remain non-causal and strong for understanding, while generation stays cleanly decoupled via an optional decoder. Across standard retrieval, VQA, and visual reasoning benchmarks, BRIDGE outperforms comparable VLMs while preserving the bi-encoder efficiency of contrastive models. We make our code publicly available at https://github.com/jfeinashley/BRIDGE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
