Transcoders 追踪视觉语言模型中的视觉基础与幻觉
Transcoders Trace Visual Grounding and Hallucinations in Vision-Language Models
AI总结：
采用基于Transcoders的功能中心框架分解视觉语言模型的计算路径，揭示视觉输入如何影响文本生成，并通过反事实分析和图结构特征预测幻觉。
摘要：
生成式视觉语言模型（VLM）在多模态推理上表现良好，但视觉输入如何转化为文本仍知之甚少。现有的VLM可解释性工作使用稀疏自编码器（SAE），其分解静态残差表示，忽略了驱动跨模态交互的功能更新。我们采用基于Transcoders的功能中心框架，Transcoders是MLP子层的稀疏近似，作为逐层计算的因果代理。应用于Gemma 3-4B-IT，该框架将模型分解为可解释的计算路径，连接图像块到文本生成中的方向。在补丁消融下，Transcoder归因对视觉基础标记产生比SAE归因更强且更稳定的效果，并与语义相关的图像区域更好对齐。假视觉基础反事实分析证实恢复的路径是视觉-语言交互特有的。最后，我们对幻觉生成进行结构分析，从Transcoder产生的电路痕迹中提取基于图的指标。基于这些机制图特征的逻辑分类器以AUC 0.68预测幻觉。这些结果表明，功能中心的电路分解为VLM中的多模态计算提供了可解释且可预测的描述。
英文摘要：
Generative Vision-Language Models (VLMs) perform well on multimodal reasoning, but how visual inputs are transformed to text remains poorly understood. Existing interpretability work on VLMs uses Sparse Autoencoders (SAEs), which decompose static residual representations and miss the functional updates that drive cross-modal interaction. We adopt a function-centric framework based on Transcoders, sparse approximations of MLP sublayers that act as a causal proxy for layer-wise computation. Applied to Gemma 3-4B-IT, the framework decomposes the model into interpretable computational pathways linking image patches to directions in token generation. Transcoder attributions produce stronger and more stable effects on visually grounded tokens under patch ablation than SAE attributions, and align better with semantically relevant image regions. A False Visual Grounding counterfactual analysis confirms that the recovered pathways are specific to vision-language interaction.Finally, we perform a structural analysis of hallucinated generations, by extracting graph-based indicators from circuit traces produced by the transcoders. A logistic classifier over these mechanistic graph features predicts hallucinations at AUC $0.68$. These results show that function-centric circuit decomposition yields interpretable and predictive accounts of multimodal computation in VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
