arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.05809cs.CV
收藏

聚焦-扫描-精炼:从人类视觉感知到高效的视觉令牌修剪

Focus-Scan-Refine: From Human Visual Perception to Efficient Visual Token Pruning

Enwei Tong, Yuanchao Bai, Yao Zhu, Junjun Jiang, Xianming Liu

AI总结:

FSR是一种受人类视觉感知启发的高效视觉令牌修剪框架,通过聚焦关键证据、全局扫描和上下文精炼,提升视觉-语言模型的准确率与效率平衡。

摘要:

视觉-语言模型(VLMs)通常生成大量视觉令牌,这显著增加了推理延迟和内存占用;而无需训练的令牌修剪提供了一种实用的解决方案,但现有方法仍难以在极端压缩下平衡局部证据和全局上下文。我们提出了Focus-Scan-Refine(FSR),一种受人类启发、即插即用的修剪框架,模仿人类回答视觉问题的方式:聚焦关键证据,如有需要则全局扫描,然后通过聚合相关细节来精炼扫描的上下文。FSR首先通过结合视觉重要性与指令相关性来聚焦关键证据,避免偏向于视觉显著但与查询无关的区域。然后,它根据聚焦集条件扫描互补的上下文,选择与聚焦证据最不同的令牌。最后,FSR通过基于相似性的分配和加权合并将附近的信息令牌聚合到扫描锚点中,而无需增加令牌预算。在多个VLM后端和视觉-语言基准测试中,FSR在现有最先进修剪方法上一致地提高了准确率-效率权衡。源代码可在https://github.com/ILOT-code/FSR找到。

英文摘要:

Vision-language models (VLMs) often generate massive visual tokens that greatly increase inference latency and memory footprint; while training-free token pruning offers a practical remedy, existing methods still struggle to balance local evidence and global context under aggressive compression. We propose Focus-Scan-Refine (FSR), a human-inspired, plug-and-play pruning framework that mimics how humans answer visual questions: focus on key evidence, then scan globally if needed, and refine the scanned context by aggregating relevant details. FSR first focuses on key evidence by combining visual importance with instruction relevance, avoiding the bias toward visually salient but query-irrelevant regions. It then scans for complementary context conditioned on the focused set, selecting tokens that are most different from the focused evidence. Finally, FSR refines the scanned context by aggregating nearby informative tokens into the scan anchors via similarity-based assignment and score-weighted merging, without increasing the token budget. Extensive experiments across multiple VLM backbones and vision-language benchmarks show that FSR consistently improves the accuracy-efficiency trade-off over existing state-of-the-art pruning methods. The source codes can be found at https://github.com/ILOT-code/FSR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑