TinyDrop:由微型模型引导的视觉Transformer令牌丢弃方法
TinyDrop: Tiny Model Guided Token Dropping for Vision Transformers
AI总结:
为降低视觉Transformer推理成本且不损精度,提出免训练的TinyDrop框架,由轻量模型引导丢弃低重要性令牌,即插即用且适配多架构,可将ViT的FLOPs最高降80%且精度损失极小。
摘要:
视觉Transformer(ViTs)在图像分类任务中表现出色,但处理所有图像令牌会带来高昂的计算成本。为在不损失精度的前提下降低大型ViT的推理成本,我们提出TinyDrop——一种由轻量视觉模型引导的免训练令牌丢弃框架。引导模型在推理过程中估算令牌重要性,从而在大型ViT模型需执行注意力计算时,选择性丢弃低重要性令牌。该框架即插即用,无需修改架构,兼容多种ViT架构。在标准图像分类基准上的评估表明,本框架可将ViT的浮点运算量(FLOPs)最高降低80%,且精度下降极小,凸显了其泛化能力及在高效ViT分类任务中的实用价值。
英文摘要:
Vision Transformers (ViTs) achieve strong performance in image classification but incur high computational costs from processing all image tokens. To reduce inference costs in large ViTs without compromising accuracy, we propose TinyDrop, a training-free token dropping framework guided by a lightweight vision model. The guidance model estimates the importance of tokens while performing inference, thereby selectively discarding low-importance tokens if large vit models need to perform attention calculations. The framework operates plug-and-play, requires no architectural modifications, and is compatible with diverse ViT architectures. Evaluations on standard image classification benchmarks demonstrate that our framework reduces FLOPs by up to 80% for ViTs with minimal accuracy degradation, highlighting its generalization capability and practical utility for efficient ViT-based classification.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。