逃避视觉失语：面向视觉-语言模型的对比自适应语义令牌修剪
Evading Visual Aphasia: Contrastive Adaptive Semantic Token Pruning for Vision-Language Models
AI总结：
本文提出COAST方法，通过对比自适应语义路由实现视觉-语言模型的高效令牌修剪，减少77.8%的视觉令牌并提升2.15倍的推理速度，同时保持98.64%的原始性能。
摘要：
在视觉-语言推理中，低注意力视觉令牌是否真正冗余？现有修剪方法通常假设如此，通过浅层文本到图像注意力对视觉令牌进行排序并丢弃低分片段以加速LVLM推理。我们证明这种标量标准在组合推理中不可靠：早期层被忽略的令牌可能在后期成为解决次级对象、空间关系和上下文线索的关键。提前修剪可能导致视觉失语，即模型失去视觉基础并依赖语言先验。我们引入COAST（COntrastive Adaptive Semantic Token Pruning），一种无需训练的修剪框架，将压缩视为自适应语义路由。COAST利用原生跨模态注意力识别查询特定的锚点并估计上下文分散情况，通过注意力熵估计上下文分散情况，然后自适应平衡语义证据与空间上下文的保留比例。它进一步使用对比路由分数来保留与锚点对齐的证据和互补的空间上下文。在七个基准测试中，COAST将视觉令牌减少77.8%，在保持98.64%的原始平均性能的同时实现了2.15倍的延迟加速。除了单一主干或压缩设置外，COAST在多种令牌预算和多个LVLM家族中均优于强大的修剪基线，表明自适应语义路由是单次标量修剪的稳健替代方案。
英文摘要：
Are low-attention visual tokens truly redundant in vision-language reasoning? Existing pruning methods often assume so, ranking visual tokens by shallow text-to-image attention and discarding low-scoring patches to accelerate LVLM inference. We show that this scalar criterion is unreliable for compositional reasoning: tokens ignored in early layers can later become essential for resolving secondary objects, spatial relations, and contextual cues. Premature pruning can therefore induce Visual Aphasia, a failure mode in which the model loses visual grounding and falls back on language priors. We introduce COAST (COntrastive Adaptive Semantic Token Pruning), a training-free pruning framework that casts compression as adaptive semantic routing. COAST uses native cross-modal attention to identify query-specific anchors and estimate contextual dispersion via attention entropy, then adapts the retention trade-off between semantic evidence and spatial context. It further uses a contrastive routing score to preserve both anchor-aligned evidence and complementary spatial context. Across seven benchmarks, COAST reduces visual tokens by 77.8% and achieves a 2.15x latency speedup while retaining 98.64% of the original average performance. Beyond a single backbone or compression setting, COAST consistently outperforms strong pruning baselines across token budgets and generalizes across multiple LVLM families, showing that adaptive semantic routing is a robust alternative to one-shot scalar pruning
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
