arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2308.01045cs.CV
收藏

用于语义分割的普通 Vision Transformer 中的动态 Token 剪枝

Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation

CSIRO(联邦科学与工业研究组织);South China University of Technology(华南理工大学);The University of Adelaide(阿德莱德大学)

Quan Tang, Bowen Zhang, Jiajun Liu, Fagui Liu, Yifan Liu

AI总结:

针对普通 Vision Transformer 语义分割计算量高的问题,提出按 token 难度提前退出并保留每类高置信度 token 的 DToP 动态剪枝方法,在不降低精度下减少 20%–35% 计算成本。

摘要:

Vision transformer 已在多种视觉任务上取得领先性能,但仍面临较高的计算复杂度。在语义分割等密集预测任务中,情况会进一步恶化,因为高分辨率输入和输出通常意味着计算中涉及更多 token。直接移除注意力较低的 token 已在图像分类任务中被讨论,但无法扩展到语义分割,因为每个 patch 都需要得到密集预测。为此,本工作提出一种基于 token 提前退出的 Dynamic Token Pruning(DToP,动态 Token 剪枝)方法,用于语义分割。受人类由粗到细的分割过程启发,我们自然地将广泛采用的、基于辅助损失的网络架构划分为若干阶段,其中每个辅助块都会评估每个 token 的难度等级。我们可以在不完成整个前向传播的情况下,提前确定简单 token 的预测。此外,我们为每个语义类别保留置信度最高的 k 个 token,以维持具有代表性的上下文信息。因此,计算复杂度会随输入难度而变化,类似于人类进行分割的方式。实验表明,所提出的 DToF 架构可为当前基于普通 vision transformer 的语义分割方法平均降低 20%–35% 的计算成本,且不会造成精度下降。

英文摘要:

Vision transformers have achieved leading performance on various visual tasks yet still suffer from high computational complexity. The situation deteriorates in dense prediction tasks like semantic segmentation, as high-resolution inputs and outputs usually imply more tokens involved in computations. Directly removing the less attentive tokens has been discussed for the image classification task but can not be extended to semantic segmentation since a dense prediction is required for every patch. To this end, this work introduces a Dynamic Token Pruning (DToP) method based on the early exit of tokens for semantic segmentation. Motivated by the coarse-to-fine segmentation process by humans, we naturally split the widely adopted auxiliary-loss-based network architecture into several stages, where each auxiliary block grades every token's difficulty level. We can finalize the prediction of easy tokens in advance without completing the entire forward pass. Moreover, we keep $k$ highest confidence tokens for each semantic category to uphold the representative context information. Thus, computational complexity will change with the difficulty of the input, akin to the way humans do segmentation. Experiments suggest that the proposed DToP architecture reduces on average $20\% - 35\%$ of computational cost for current semantic segmentation methods based on plain vision transformers without accuracy degradation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑