arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.15033cs.CL
收藏

SmartTrim:面向高效视觉-语言模型的自适应令牌与注意力剪枝

SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Models

Harbin Institute of Technology(哈尔滨工业大学);ETH Zurich(苏黎世联邦理工学院);Du Xiaoman (Beijing) Science Technology Co., Ltd(度小满(北京)科技有限公司);Peng Cheng Laboratory(鹏城实验室)

Zekun Wang, Jingchang Chen, Wangchunshu Zhou, Haichao Zhu, Jiafeng Liang, Liping Shan, Ming Liu, Dongliang Xu, Qing Yang, Bing Qin

AI总结:

本文提出SmartTrim自适应加速框架,通过轻量级模块逐层剪枝冗余令牌和注意力头,并结合自蒸馏保持预测一致性,在多种视觉-语言任务上实现2-3倍加速且性能损失极小。

摘要:

尽管基于Transformer的视觉-语言模型(VLMs)在各种视觉-语言任务上取得了显著性能,但它们在输入和参数方面存在冗余,严重阻碍了其在实际应用中的效率。此外,令牌表示和模型参数(如注意力头)中的冗余程度对于不同输入差异很大。针对这些挑战,我们提出了SmartTrim,一个面向VLMs的自适应加速框架,它按实例调整计算开销。具体而言,我们在原始骨干网络中集成轻量级模块,以识别并剪枝每一层中冗余的令牌表示和注意力头。此外,我们设计了一种自蒸馏策略,以增强剪枝后模型与其全容量对应模型预测之间的一致性。在多种视觉-语言任务上的实验结果表明,SmartTrim将原始模型加速2-3倍,同时性能下降极小,凸显了与先前方法相比的有效性和效率。代码将在https://github.com/kugwzk/SmartTrim提供。

英文摘要:

Despite achieving remarkable performance on various vision-language tasks, Transformer-based Vision-Language Models (VLMs) suffer from redundancy in inputs and parameters, significantly hampering their efficiency in real-world applications. Moreover, the degree of redundancy in token representations and model parameters, such as attention heads, varies significantly for different inputs. In light of the challenges, we propose SmartTrim, an adaptive acceleration framework for VLMs, which adjusts the computational overhead per instance. Specifically, we integrate lightweight modules into the original backbone to identify and prune redundant token representations and attention heads within each layer. Furthermore, we devise a self-distillation strategy to enhance the consistency between the predictions of the pruned model and its fully-capacity counterpart. Experimental results across various vision-language tasks consistently demonstrate that SmartTrim accelerates the original model by 2-3 times with minimal performance degradation, highlighting the effectiveness and efficiency compared to previous approaches. Code will be available at https://github.com/kugwzk/SmartTrim.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑