VisionZip: 更长并非必要，但在视觉语言模型中更优
VisionZip: Longer is Better but Not Necessary in Vision Language Models
AI总结：
本文提出VisionZip方法，通过选择信息性视觉token提升效率并保持性能，实验显示在多种设置下性能提升至少5%，同时显著加快推理速度。
摘要：
最近视觉语言模型的进步通过增加视觉token长度来提升性能，使它们比文本token更长且显著增加计算成本。然而，我们发现流行视觉编码器（如CLIP和SigLIP）生成的视觉token存在显著冗余。为此，我们引入VisionZip，一种简单而有效的方法，通过选择信息性token输入语言模型，减少视觉token冗余，提高效率并保持模型性能。所提的VisionZip可广泛应用于图像和视频理解任务，并适用于现实场景中的多轮对话，其中先前方法表现不佳。实验结果表明，VisionZip在几乎所有设置中均优于先前的最先进方法，至少提升5%的性能。此外，我们的方法显著提高了模型推理速度，将预填充时间提高8倍，并使LLaVA-Next 13B模型的推理速度比LLaVA-Next 7B模型更快，同时取得更好的结果。此外，我们分析了冗余的原因，并鼓励社区关注提取更好的视觉特征，而不是仅仅增加token长度。我们的代码可在https://github.com/dvlab-research/VisionZip上获得。
英文摘要：
Recent advancements in vision-language models have enhanced performance by increasing the length of visual tokens, making them much longer than text tokens and significantly raising computational costs. However, we observe that the visual tokens generated by popular vision encoders, such as CLIP and SigLIP, contain significant redundancy. To address this, we introduce VisionZip, a simple yet effective method that selects a set of informative tokens for input to the language model, reducing visual token redundancy and improving efficiency while maintaining model performance. The proposed VisionZip can be widely applied to image and video understanding tasks and is well-suited for multi-turn dialogues in real-world scenarios, where previous methods tend to underperform. Experimental results show that VisionZip outperforms the previous state-of-the-art method by at least 5% performance gains across nearly all settings. Moreover, our method significantly enhances model inference speed, improving the prefilling time by 8x and enabling the LLaVA-Next 13B model to infer faster than the LLaVA-Next 7B model while achieving better results. Furthermore, we analyze the causes of this redundancy and encourage the community to focus on extracting better visual features rather than merely increasing token length. Our code is available at https://github.com/dvlab-research/VisionZip .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
