信息正则化注意力用于以视觉为中心的推理
Information-Regularized Attention for Visual-Centric Reasoning
AI总结：
针对视觉语言模型中的对象幻觉、视觉基础弱和灾难性遗忘问题，提出信息正则化注意力机制，通过随机注意力显式调控视觉信息注入，改善表示学习稳定性。
摘要：
视觉语言模型已成为多模态学习的范式，但由于对象幻觉、视觉基础弱以及全参数指令调优后的灾难性遗忘，仍然不稳定。我们认为这些失败源于在标准的下一个词预测目标中缺乏对视觉表示学习的显式控制。因此，视觉嵌入被被动优化，容易注入冗余或虚假信号。为了解决这个问题，我们引入了信息正则化注意力，一种随机注意力机制，显式调控注入中间Transformer层隐藏状态的视觉信息量。这种局部重参数化将视觉表示的不确定性转化为跨数据点独立的局部噪声。除了评估模型性能，我们还量化了嵌入属性，其中IRA在所有层中产生更平滑的曲率轨迹并抑制注意力汇聚，表明视觉信号的变换更加稳定。我们的结果表明，随机注意力不仅是一种正则化器，而且是生成架构中表示学习的关键贡献者，为构建更可靠的视觉语言模型提供了新方向。
英文摘要：
Vision-language models (VLMs) have become a paradigm for multimodal learning, yet remain unstable due to object hallucination, weak visual grounding, and catastrophic forgetting after full-parameter instruction tuning. We claim these failures result from a lack of explicit control over visual representation learning during the standard next-token prediction objective. As a result, visual embeddings thus become passively optimized and prone to injecting redundant or spurious signals. To counter this, we introduce Information-Regularized Attention (IRA), a stochastic attention mechanism that explicitly regulates the amount of visual information injected into the hidden states of intermediate transformer layers. This local reparameterization translates uncertainty about visual representations into local noise that is independent across data points. Beyond evaluating model performance, we also quantify embedding properties, where IRA produces smoother curvature trajectories and suppresses attention-sink across all layers, indicating a more stable transformation of the visual signal. Our results suggest that stochastic attention is not merely a regularizer but a key contributor to representation learning in a generative architecture, offering a new direction for building more reliable VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
