感知定位的 Token 剪枝：从剪枝导致的视觉定位性能急剧下降中恢复
Grounding-Aware Token Pruning: Recovering from Drastic Performance Drops in Visual Grounding Caused by Pruning
AI总结：
提出 Grounding-Aware Token Pruning (GAP) 方法，通过调整位置 ID 解决剪枝导致的视觉定位性能下降问题，无需额外开销即可恢复大部分性能。
摘要：
近期的多模态大语言模型（MLLMs）在视觉定位方面表现出色，已成为各种视觉-语言应用的通用接口。这一进展推动了 token 剪枝方法的发展，以缓解处理大量视觉 token 带来的高昂计算成本。然而，我们观察到剪枝显著削弱了模型的定位能力，导致预测错误和性能急剧下降。例如，在指代表达理解（REC）任务中，剪枝导致 LLaVA 在 RefCOCO 验证集上的准确率从 56.14% 降至 15.34%。我们的分析发现，剪枝后位置 ID 的错位是性能下降的主要原因，因为这些 ID 的顺序和数值对维持定位任务性能至关重要。为解决此问题，我们提出了 Grounding-Aware Token Pruning (GAP)，这是一种简单有效的位置 ID 调整方法，能将 REC 准确率恢复至 51.42%，相当于无剪枝设定下原始性能的 90%，且无需额外训练、内存或计算开销。应用于 Shikra、MiniGPTv2 和 LLaVA 系列等模型时，我们的方法在各种 token 剪枝策略中均能持续提升性能。
英文摘要：
Recent Multimodal Large Language Models (MLLMs) have demonstrated strong performance in visual grounding, establishing themselves as a general interface for various vision-language applications. This progress has driven the development of token pruning methods to mitigate the high computational costs associated with processing numerous visual tokens. However, we observe that pruning significantly weakens the model's grounding ability, leading to incorrect predictions and drastic performance degradation. In Referring Expression Comprehension (REC), for instance, pruning causes the accuracy of LLaVA on the RefCOCO validation set to drop from 56.14% to 15.34%. Our analysis identifies misaligned position IDs after pruning as the primary cause of this degradation, as both the order and value of these IDs are crucial for maintaining performance in grounding tasks. To address this issue, we propose Grounding-Aware Token Pruning (GAP), a simple yet effective adjustment to position IDs that recovers REC accuracy back to 51.42%, which is 90% of the original performance in the without pruning setting, all while requiring no additional training, memory, or computational overhead. Applied to models such as Shikra, MiniGPTv2, and the LLaVA series, our method consistently improves performance across various token pruning strategies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
