Fast-iTPN：具有 Token Migration 的整体预训练 Transformer 金字塔网络
Fast-iTPN: Integrally Pre-Trained Transformer Pyramid Network with Token Migration
AI总结：
本文提出Fast-iTPN，通过联合优化主干与颈部缩小迁移差距，并引入Token Migration和Token Gathering机制大幅降低内存开销与加速推理，在多项视觉任务上取得SOTA性能且几乎无精度损失。
摘要：
我们提出了整体预训练 transformer 金字塔网络（iTPN），旨在联合优化网络主干和颈部，使得表示模型与下游任务之间的迁移差距最小化。iTPN 天生具备两项精心设计：1）首个基于视觉 transformer（ViT）的预训练特征金字塔。2）使用掩蔽特征建模（MFM）对特征金字塔进行多阶段监督。iTPN 升级为 Fast-iTPN，通过两项灵活设计降低计算内存开销并加速推理。1）Token migration：丢弃主干中冗余的 token，并在无注意力操作的情况下于特征金字塔中补充它们。2）Token gathering：通过引入少量 gathering token 来减少全局注意力造成的计算成本。base/large 级别的 Fast-iTPN 在 ImageNet-1K 上取得 88.75%/89.5% 的 top-1 准确率。在使用 DINO 的 1x 训练计划下，base/large 级别的 Fast-iTPN 在 COCO 目标检测上取得 58.4%/58.8% 的 box AP，并在使用 MaskDINO 的 ADE20K 语义分割上取得 57.5%/58.7% 的 mIoU。Fast-iTPN 能将推理过程加速高达 70%，且性能损失可忽略不计，展现了其作为下游视觉任务强大主干的潜力。代码可在 github.com/sunsmarterjie/iTPN 获取。
英文摘要：
We propose integrally pre-trained transformer pyramid network (iTPN), towards jointly optimizing the network backbone and the neck, so that transfer gap between representation models and downstream tasks is minimal. iTPN is born with two elaborated designs: 1) The first pre-trained feature pyramid upon vision transformer (ViT). 2) Multi-stage supervision to the feature pyramid using masked feature modeling (MFM). iTPN is updated to Fast-iTPN, reducing computational memory overhead and accelerating inference through two flexible designs. 1) Token migration: dropping redundant tokens of the backbone while replenishing them in the feature pyramid without attention operations. 2) Token gathering: reducing computation cost caused by global attention by introducing few gathering tokens. The base/large-level Fast-iTPN achieve 88.75%/89.5% top-1 accuracy on ImageNet-1K. With 1x training schedule using DINO, the base/large-level Fast-iTPN achieves 58.4%/58.8% box AP on COCO object detection, and a 57.5%/58.7% mIoU on ADE20K semantic segmentation using MaskDINO. Fast-iTPN can accelerate the inference procedure by up to 70%, with negligible performance loss, demonstrating the potential to be a powerful backbone for downstream vision tasks. The code is available at: github.com/sunsmarterjie/iTPN.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
