arXiv 2602.236992026-02-27cs.CVcs.CL

HiDrop：通过晚期注入、凹形金字塔剪枝和早期退出实现MLLM中的层次视觉令牌减少

HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit

Institute of Digital Twin, Eastern Institute of Technology（数字孪生研究所，东部技术研究所）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波空间智能与数字衍生关键实验室）；University of Science and Technology of China（中国科学技术大学）；Shanghai Jiao Tong University（上海交通大学）；Munich Center for Machine Learning, LMU Munich（慕尼黑大学机器学习中心，慕尼黑大学）

Hao Wu, Yingqi Fan, Jinyang Dai, Junlong Tong, Yunpu Ma, Xiaoyu Shen

备注：: Accepted to ICLR 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

HiDrop通过晚期注入、凹形金字塔剪枝和早期退出机制，实现多模态大语言模型中视觉令牌的高效减少，提升训练效率并保持性能。

摘要：

多模态大语言模型（MLLMs）中处理视觉令牌的二次计算成本阻碍了其广泛应用。尽管渐进式视觉令牌剪枝提供了一个有前景的解决方案，但当前方法误判了浅层层的功能并使用刚性的调度方案，无法释放全部效率潜力。为了解决这些问题，我们提出了HiDrop框架，该框架将令牌剪枝与MLLM层的真实层次功能对齐。HiDrop有两个关键创新：（1）晚期注入，通过绕过被动浅层层，在活跃融合开始的地方引入视觉令牌；（2）带有早期退出机制的凹形金字塔剪枝，以动态调整中间和深层层的剪枝率。此过程通过层间相似性度量和可微的top-k操作符进行优化。为确保实际效率，HiDrop进一步结合了持久的位置编码、与FlashAttention兼容的令牌选择以及并行解耦的视觉计算，以消除与动态令牌减少相关的隐藏开销。广泛的实验表明，HiDrop在压缩约90%的视觉令牌的同时保持原始性能，并将训练速度提升1.72倍。我们的工作不仅为高效MLLM训练和推理设定了新的最先进的标准，还为多模态融合的层次结构提供了有价值的见解。代码已发布在https://github.com/EIT-NLP/HiDrop。

英文摘要：

The quadratic computational cost of processing vision tokens in Multimodal Large Language Models (MLLMs) hinders their widespread adoption. While progressive vision token pruning offers a promising solution, current methods misinterpret shallow layer functions and use rigid schedules, which fail to unlock the full efficiency potential. To address these issues, we propose HiDrop, a framework that aligns token pruning with the true hierarchical function of MLLM layers. HiDrop features two key innovations: (1) Late Injection, which bypasses passive shallow layers to introduce visual tokens exactly where active fusion begins; and (2) Concave Pyramid Pruning with an Early Exit mechanism to dynamically adjust pruning rates across middle and deep layers. This process is optimized via an inter-layer similarity measure and a differentiable top-k operator. To ensure practical efficiency, HiDrop further incorporates persistent positional encoding, FlashAttention-compatible token selection, and parallel decoupling of vision computation to eliminate hidden overhead associated with dynamic token reduction. Extensive experiments show that HiDrop compresses about 90% visual tokens while matching the original performance and accelerating training by 1.72 times. Our work not only sets a new state-of-the-art for efficient MLLM training and inference but also provides valuable insights into the hierarchical nature of multimodal fusion. The code is released at https://github.com/EIT-NLP/HiDrop.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。