HERO：重新思考高分辨率大型视觉-语言模型中的视觉 Token 早期丢弃
HERO: Rethinking Visual Token Early Dropping in High-Resolution Large Vision-Language Models
AI总结：
针对高分辨率大型视觉-语言模型视觉 token 过多的问题，提出无需训练的 HERO 早期丢弃框架，结合自适应预算分配与功能感知选择以提升效率并保持精度。
摘要：
通过将高分辨率图像裁剪为局部图块并独立编码，高分辨率大型视觉-语言模型（HR-LVLM）已展现出卓越的细粒度视觉理解能力。然而，这种分治范式显著增加了视觉 token 数量，导致大量计算和内存开销。为更好地理解并应对这一挑战，我们通过实证研究 HR-LVLM 中视觉 token 的利用情况，并揭示三个关键发现：（1）局部图块具有不同的重要性，这由视觉显著性和任务相关性共同决定；（2）基于 CLIP 的视觉编码器中的 CLS token 在各层间表现出两阶段注意力模式，每个阶段关注不同类型的视觉 token；（3）不同阶段被强调的视觉 token 编码不同粒度级别的信息，在 LVLM 中发挥互补作用。基于这些洞见，我们提出 HERO，这是一个高分辨率视觉 token 早期丢弃框架，融合了内容自适应的 token 预算分配与功能感知的 token 选择。通过准确估计图块级重要性并有选择地保留具有互补作用的视觉 token，HERO 以无需训练的方式，在多种基准和模型规模上实现了更优的效率-准确性权衡。本研究为 HR-LVLM 的高效推理提供了实证洞见和实用解决方案。
英文摘要：
By cropping high-resolution images into local tiles and encoding them independently, High-Resolution Large Vision-Language Models (HR-LVLMs) have demonstrated remarkable fine-grained visual understanding capabilities. However, this divide-and-conquer paradigm significantly increases the number of visual tokens, resulting in substantial computational and memory overhead. To better understand and address this challenge, we empirically investigate visual token utilization in HR-LVLMs and uncover three key findings: (1) the local tiles have varying importance, jointly determined by visual saliency and task relevance; (2) the CLS token in CLIP-based vision encoders exhibits a two-stage attention pattern across layers, with each stage attending to different types of visual tokens; (3) the visual tokens emphasized at different stages encode information at varying levels of granularity, playing complementary roles within LVLMs. Building on these insights, we propose HERO, a High-resolution visual token early dropping framework that integrates content-adaptive token budget allocation with function-aware token selection. By accurately estimating tile-level importance and selectively retaining visual tokens with complementary roles, HERO achieves superior efficiency-accuracy trade-offs across diverse benchmarks and model scales, all in a training-free manner. This study provides both empirical insights and practical solutions toward efficient inference in HR-LVLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
