重新思考视觉-语言推理中的因果掩码注意力
Rethinking Causal Mask Attention for Vision-Language Inference
AI总结：
针对视觉-语言推理中因果掩码的刚性限制，提出未来感知注意力家族，通过池化聚合未来视觉上下文到过去表示，提升推理性能。
摘要：
因果注意力已成为自回归视觉-语言模型（VLMs）的基础机制，将文本和视觉输入统一在单一生成框架下。然而，现有的基于因果掩码的策略是从大型语言模型（LLMs）中继承而来的，这些策略是为纯文本解码而设计的，其对视觉标记的适配在预填充阶段未得到充分解决。对视觉查询的未来位置进行严格掩码引入了过于刚性的约束，阻碍了模型利用未来上下文的能力，而未来上下文通常包含对准确推理至关重要的语义线索。在这项工作中，我们实证研究了不同的因果掩码策略如何影响视觉-语言推理，并针对这一场景提出了一系列未来感知注意力机制。我们首先实证分析了预览未来标记对视觉查询的影响，并证明刚性掩码削弱了模型捕获有用上下文语义表示的能力。基于这些发现，我们提出了一种轻量级注意力家族，通过池化将未来视觉上下文聚合到过去表示中，有效保持自回归结构同时增强跨标记依赖。我们在多种视觉-语言推理设置中评估了一系列因果掩码，并表明选择性地将未来语义上下文压缩到过去表示中有益于推理。
英文摘要：
Causal attention has become a foundational mechanism in autoregressive vision-language models (VLMs), unifying textual and visual inputs under a single generative framework. However, existing causal mask-based strategies are inherited from large language models (LLMs) where they are tailored for text-only decoding, and their adaptation to vision tokens is insufficiently addressed in the prefill stage. Strictly masking future positions for vision queries introduces overly rigid constraints, which hinder the model's ability to leverage future context that often contains essential semantic cues for accurate inference. In this work, we empirically investigate how different causal masking strategies affect vision-language inference and then propose a family of future-aware attentions tailored for this setting. We first empirically analyze the effect of previewing future tokens for vision queries and demonstrate that rigid masking undermines the model's capacity to capture useful contextual semantic representations. Based on these findings, we propose a lightweight attention family that aggregates future visual context into past representations via pooling, effectively preserving the autoregressive structure while enhancing cross-token dependencies. We evaluate a range of causal masks across diverse vision-language inference settings and show that selectively compressing future semantic context into past representations benefits the inference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
