发表机构
Peking University; Xiaomi Corporation; The University of Hong Kong（北京大学; 小米公司; 香港大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对多模态LLM中混合ViT注意力设计不足的问题，提出语义头专业化（SHS）概念，据此设计Ariadne注意力机制，在22项图像视频任务上性能与全注意力相当，计算量降低6.5倍。
AI 中文摘要
混合注意力机制在前沿大语言模型（LLM）中占据主导地位，但多模态LLM中的视觉Transformer（ViT）缺乏令人满意的混合设计，且尚未明确为何某些注意力模式效果更佳。为填补这一空白，我们研究ViT注意力头，发现它们分化为物体专家和背景专家角色，该模式在全注意力下最为显著，我们将此称为语义头专业化（Semantic Head Specialization, SHS）。我们提出SHS指数（SHS-Index）以量化这种专业化，证明其可区分全注意力与分块窗口ViT，且与下游基准性能高度相关。随后，我们确定了影响SHS的三个结构因素：窗口交互、令牌序列化和局部Softmax分配，并将其作为混合注意力的设计原则。基于这些因素，我们设计了Ariadne注意力机制，该混合机制在22项图像和视频任务上的表现与全注意力相当，但注意力计算量降低6.5倍。我们的研究结果表明，头专业化是可测量的属性，可用于在多模态LLM规模下诊断和设计原理性混合ViT注意力机制。
英文摘要
Hybrid attention dominates frontier LLMs, yet Vision Transformers (ViTs) in multimodal LLMs lack a satisfactory hybrid design, with no consensus on why certain attention patterns work better. To fill this gap, we study ViT attention heads and find they differentiate into object- and background-specialist roles, a pattern most pronounced under full attention; we call this Semantic Head Specialization (SHS). We propose SHS-Index to quantify this specialization, show that it distinguishes full-attention from chunk-window ViTs, and find that it strongly tracks downstream benchmark performance. We then identify three structural factors that shape SHS---window interaction, token serialization, and local softmax allocation---and use them as design principles for hybrid attention. Guided by these factors, we design Ariadne Attention, a hybrid that matches full attention on 22 image and video tasks at 6.5x less attention compute. Our findings establish head specialization as a measurable property for diagnosing and designing principled hybrid ViT attention at the multimodal-LLM scale.
