别再在多模态语言模型中寻找重要 Token：重复度更关键
Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More
AI总结：
本文提出无需训练的 DART，依据视觉 token 间的重复度而非重要性进行剪枝，在剪除 88.9% token 时保持性能，并带来最高 2.99 倍加速。
摘要：
多模态大语言模型中的视觉 token 因其长度远超语言模态，往往造成巨大的计算开销。近期大量方法试图通过 token 剪枝来解决这一问题：它们首先为 token 定义重要性准则，然后在推理过程中剪除不重要的视觉 token。然而，本文表明，重要性并不是决定某个 token 是否应被剪枝的理想指标。令人惊讶的是，这种方法通常比随机 token 剪枝产生更差的性能，并且会导致与高效注意力计算算子不兼容。相反，我们提出了 DART（Duplication-Aware Reduction of Tokens，重复感知 token 缩减），它根据 token 与其他 token 之间的重复度进行剪枝，从而实现显著且无需训练的加速。具体而言，DART 选择一小部分枢轴 token，然后保留与这些枢轴重复度较低的 token，以确保 token 剪枝过程中的信息损失最小。实验表明，DART 能够剪除 88.9% 的视觉 token，同时保持相当的性能，使总时间和预填充阶段分别实现 1.99 倍和 2.99 倍的加速，并与高效注意力算子具有良好的兼容性。我们的代码可在 https://github.com/ZichenWen1/DART 获取。
英文摘要：
Vision tokens in multimodal large language models often dominate huge computational overhead due to their excessive length compared to linguistic modality. Abundant recent methods aim to solve this problem with token pruning, which first defines an importance criterion for tokens and then prunes the unimportant vision tokens during inference. However, in this paper, we show that the importance is not an ideal indicator to decide whether a token should be pruned. Surprisingly, it usually results in inferior performance than random token pruning and leading to incompatibility to efficient attention computation operators.Instead, we propose DART (Duplication-Aware Reduction of Tokens), which prunes tokens based on its duplication with other tokens, leading to significant and training-free acceleration. Concretely, DART selects a small subset of pivot tokens and then retains the tokens with low duplication to the pivots, ensuring minimal information loss during token pruning. Experiments demonstrate that DART can prune 88.9% vision tokens while maintaining comparable performance, leading to a 1.99$\times$ and 2.99$\times$ speed-up in total time and prefilling stage, respectively, with good compatibility to efficient attention operators. Our codes are available at https://github.com/ZichenWen1/DART.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
