线性机制用于视觉语言模型中的时空推理
Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models
AI总结：
本文提出了一种线性机制用于视觉语言模型中的时空推理，通过分析空间ID和时间ID的结合，揭示了模型内部的因果推理过程，以提升模型的可解释性和设计能力。
摘要：
时空推理是视觉语言模型（VLMs）的显著能力，但其背后机制仍然大多模糊。我们假设在VLM计算中，视觉/几何和文本表示的空结构必须在某处结合。我们寻找这种融合，并询问所识别的表示是否能通过线性模型因果解释输入输出模型行为的某些方面。我们通过实证表明，VLMs通过将空间ID线性绑定到文本激活来编码物体位置，然后通过语言标记进行推理。通过严格的因果干预，我们证明这些在模型中普遍存在ID可以在中间VLM层系统地调解模型信念。此外，我们发现空间ID可以作为诊断工具，用于识别现有VLMs的局限性，并作为有价值的训练信号。我们还将分析扩展到视频VLMs，并识别出类似的线性时间ID机制。通过表征我们提出的时空ID机制，我们阐明了VLMs中之前未被充分探索的内部推理过程，朝着改进可解释性和更对齐和有能力模型的原理性设计迈进。我们发布了代码以确保可重复性：https://github.com/Raphoo/linear-mech-vlms。
英文摘要：
Spatio-temporal reasoning is a remarkable capability of Vision Language Models (VLMs), but the underlying mechanisms of such abilities remain largely opaque. We postulate that visual/geometrical and textual representations of spatial structure must be combined at some point in VLM computations. We search for such confluence, and ask whether the identified representation can causally explain aspects of input-output model behavior through a linear model. We show empirically that VLMs encode object locations by linearly binding \textit{spatial IDs} to textual activations, then perform reasoning via language tokens. Through rigorous causal interventions we demonstrate that these IDs, which are ubiquitous across the model, can systematically mediate model beliefs at intermediate VLM layers. Additionally, we find that spatial IDs serve as a diagnostic tool for identifying limitations in existing VLMs, and as a valuable learning signal. We extend our analysis to video VLMs and identify an analogous linear temporal ID mechanism. By characterizing our proposed spatiotemporal ID mechanism, we elucidate a previously underexplored internal reasoning process in VLMs, toward improved interpretability and the principled design of more aligned and capable models. We release our code for reproducibility: https://github.com/Raphoo/linear-mech-vlms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
