arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.25075cs.AI
收藏

视觉语言模型中的稀疏视觉思维电路

Sparse Visual Thought Circuits in Vision-Language Models

Yunpeng Zhou

AI总结:

研究探讨了稀疏自编码器特征在推理中的模块化假设,发现干预任务选择性特征集能小幅提升推理准确率,但联合干预则导致预测漂移和精度下降,揭示了SAE特征组合的边界。

摘要:

稀疏自编码器(SAEs)提升了多模态模型的可解释性,但其特征是否形成模块化、可组合的推理单元仍存疑问。本文测试这一假设,发现干预任务选择性特征集可小幅提升推理准确率,而联合干预则导致预测漂移和精度下降,即使在规范匹配扰动下亦然。这种非模块化电路干扰与共享内部路径中特征联合放大激活变化一致。我们开发了可复现的因果管道,以本地化和测试Qwen3-VL-8B中的稀疏视觉思维电路。在受控合成基准测试中,线性探针识别出中层解码器位置的任务类型信息。我们训练SAEs于该层,通过显式规则构建任务选择性集合,并在推理时间进行缩放和消融实验,同时量化准确率和漂移。我们的发现通过抽样子样本和排列控制验证,并在多个VLM家族和五种多样化数据集中得到复制,明确了SAE特征组合的边界,并提供了一个更可靠的VLM控制诊断框架。

英文摘要:

Sparse autoencoders (SAEs) improve interpretability in multimodal models, but it remains unclear whether SAE features form modular, composable units for reasoning-an assumption underlying many intervention-based steering methods. We test this modularity hypothesis and find it often fails: intervening on a task-selective feature set can modestly improve reasoning accuracy, while intervening on the union of two such sets reliably induces output drift (large unintended changes in predictions) and degrades accuracy, even under norm-matched perturbations. This non modular circuit interference is consistent with shared internal pathways where feature unions amplify activation shifts. We develop a reproducible causal pipeline to localize and test these sparse visual thought circuits in Qwen3-VL-8B. On a controlled synthetic benchmark with seven task types and three difficulty levels, linear probes identify a mid decoder locus for task type information. We train SAEs at this layer, construct task-selective sets via an explicit rule, and perform inference time scaling and ablation while quantifying accuracy and drift. Our findings-validated with bootstrapped subsamples and permutation controls, and replicated across multiple VLM families and five diverse datasets clarify the boundaries of SAE feature composability and provide a rigorous diagnostic framework for more reliable VLM control.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑