注视头：视觉语言模型如何观察它们所描述的内容
Gaze Heads: How VLMs Look at What They Describe
AI总结：
发现视觉语言模型的语言骨干中存在一组“注视头”，其注意力跟踪当前描述的图像区域，通过干预这些头可精确控制模型描述内容，准确率达83.1%。
摘要：
视觉语言模型在内部如何解决描述图像的任务远非显而易见。我们发现模型为此发展出一种特定机制：其语言模型骨干中的一小部分注意力头（我们称之为注视头），其注意力跟踪模型当前正在描述的图像区域。我们通过简单的相关性得分从几次前向传播中发现了它们，使用连环漫画作为受控测试平台，其中叙事顺序在空间上展开。这些注视头不仅跟踪正在描述的图像标记：将它们的注意力重定向到所选区域会强制视觉语言模型描述该区域。对前100个注视头（少于所有头的9%）进行单次注意力掩码干预，以83.1%的准确率将模型的答案引导到任何选定的漫画面板，而对随机头进行相同干预则无法重定向答案，并且对所有头进行干预会破坏生成。相同的杠杆还扩展到连续控制：在生成过程中切换注视目标会使模型在几个标记内结束当前面板描述并转向新面板。在漫画之外，相同的干预将答案重定向到自然COCO图像中的选定区域。该机制进一步在2B到32B参数的模型大小以及其他视觉语言模型架构中重复出现，尽管一些冻结编码器系列没有显示可比较的头集。更广泛地说，这表明通过机制分析识别的目标编辑可以作为实用的推理时杠杆来引导多模态模型行为，而无需任何重新训练。我们的代码、交互式演示和数据集可在以下网址获取：此 https URL
英文摘要：
How a vision-language model internally solves the task of describing an image is far from obvious. We find that the model develops a specific mechanism for this: a small set of attention heads in its language-model backbone, which we call gaze heads, whose attention tracks the image region the model is currently describing. We find them with a simple correlation score from a few forward passes, using comic strips as a controlled testbed where narrative order is laid out spatially. These gaze heads do not just track the image tokens being described: redirecting their attention to a chosen region forces the VLM to describe that region instead. A single attention-mask intervention on the top-100 gaze heads, fewer than 9% of all heads, steers the model's answer to any chosen comic panel at 83.1% accuracy, while the same intervention on random heads fails to redirect the answer, and intervening on all heads destroys generation. The same lever also extends to continuous control: switching the gaze target mid-generation makes the model wrap up its current panel description and move to the new one within a few tokens. Beyond comics, the same intervention redirects answers to chosen regions in natural COCO images. The mechanism further recurs across model sizes from 2B to 32B parameters and across other VLM architectures, although some frozen-encoder families show no comparable head set. More broadly, this shows that targeted edits identified through mechanistic analysis can serve as practical inference-time levers for steering multimodal model behavior, without any retraining. Our code, interactive demo, and datasets are available at https://gaze.baulab.info/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
