当1+1适得其反：重新思考样本感知的视觉令牌缩减的相关性-多样性平衡
When 1+1 Backfires: Rethinking Relevance-Diversity Balancing for Sample-Aware Visual Token Reduction
AI总结：
该研究针对视觉语言模型的视觉令牌压缩，提出样本感知的SAVER框架，通过几何意图先验推断样本特定的相关性-多样性权重，在推理时实现平衡，以更少令牌保留高性能且可泛化。
摘要：
视觉令牌压缩已成为降低视觉语言模型推理成本的关键技术。现有方法基于文本相关性、视觉多样性或两者的固定组合来选择令牌——将相关性-多样性平衡视为与输入无关的通用常数。我们进行系统分析以挑战这一假设。在数据集层面，对每个基准搜索最优相关性-多样性权重，发现存在显著的跨任务差异。更关键的是，我们进一步最大化每个样本的答案似然以恢复样本级的最优权重，发现其分布不仅在不同基准间差异显著，且在同一基准内也存在高方差。基于这些观察，我们提出SAVER（Sample-Aware relevance-diversity balanced Visual tokEn Reduction，样本感知相关性-多样性平衡视觉令牌缩减）框架。我们首先从文本相关令牌的空间集中度推导几何意图先验（Geometric Intent Prior，GIP）以推断样本特定的相关性-多样性权重，然后应用该权重通过加权文本质心（Weighted Text Centroid，WTC）进行相关性评分，以及通过视觉边际覆盖增益（Visual Marginal Coverage Gain，VMCG）进行多样性计算，以联合选择令牌。SAVER无需额外训练，完全在推理时实现样本感知平衡。大量实验表明，SAVER在不同基准上始终优于最先进的基线，在LLaVA-1.5-7B上仅保留11.1%的原始视觉令牌，即可保留完整模型96.5%的性能，且进一步可泛化到LLaVA-NeXT和Mini-Gemini。
英文摘要：
Visual token compression has emerged as a critical technique for reducing the inference cost of Vision-Language Models. Existing methods select tokens based on text relevance, visual diversity, or a fixed combination of both---treating the relevance-diversity balance as a universal constant independent of the input. We conduct a systematic analysis to challenge this assumption. At the dataset level, grid-searching the optimal relevance-diversity weight per benchmark reveals substantial cross-task variation. More critically, we further maximize per-sample answer likelihood to recover sample-level oracle weights, finding that their distributions not only differ markedly across benchmarks but also exhibit high within-benchmark variance. Based on these observations, we propose SAVER, a framework for Sample-Aware relevance-diversity balanced Visual tokEn Reduction. We first derive a Geometric Intent Prior (GIP) from the spatial concentration of text-relevant tokens to infer a sample-specific relevance-diversity weight, and then apply this weight to jointly select tokens via a Weighted Text Centroid (WTC) for relevance scoring and a Visual Marginal Coverage Gain (VMCG) for diversity. SAVER requires no additional training and achieves sample-aware balancing entirely at inference time. Extensive experiments demonstrate that SAVER consistently outperforms state-of-the-art baselines across diverse benchmarks, retaining 96.5% of full-model performance on LLaVA-1.5-7B with only 11.1% of the original visual tokens, and further generalizing to LLaVA-NeXT and Mini-Gemini.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
