DropPos:通过重建被丢弃位置预训练视觉Transformer
DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions
AI总结:
DropPos提出一种自监督预训练任务,通过随机丢弃位置嵌入并仅依据图像块外观重建其位置,同时引入位置平滑与注意力重建策略,显著增强了视觉Transformer的位置感知能力,在下游任务中优于有监督预训练并达到领先水平。
摘要:
由于经验观察到视觉Transformer(ViTs)对输入token的顺序相当不敏感,因此需要一种合适的自监督前置任务来增强ViTs的位置感知能力,这一需求正变得日益明显。为了解决这一问题,我们提出了DropPos,这是一种新颖的前置任务,旨在重建被丢弃的位置(Dropped Positions)。DropPos的构建方式很简单:我们首先丢弃一大部分随机的位置嵌入,然后模型仅根据每个非重叠图像块的外观,在所有可能的位置中对其实际位置进行分类。为了避免平凡解,我们通过只保留一部分可见的图像块来增加该任务的难度。此外,考虑到可能存在外观相似的不同图像块,我们提出了位置平滑和注意力重建策略来放宽这一分类问题,因为在这些情况下没有必要重建它们的精确位置。对DropPos的实证评估显示了其强大的能力。DropPos优于有监督预训练,并在广泛的下游基准上取得了与最先进的自监督方法相比具有竞争力的结果。这表明,像DropPos这样明确鼓励空间推理能力,确实有助于提高ViTs的位置感知能力。代码公开于https://github.com/Haochen-Wang409/DropPos。
英文摘要:
As it is empirically observed that Vision Transformers (ViTs) are quite insensitive to the order of input tokens, the need for an appropriate self-supervised pretext task that enhances the location awareness of ViTs is becoming evident. To address this, we present DropPos, a novel pretext task designed to reconstruct Dropped Positions. The formulation of DropPos is simple: we first drop a large random subset of positional embeddings and then the model classifies the actual position for each non-overlapping patch among all possible positions solely based on their visual appearance. To avoid trivial solutions, we increase the difficulty of this task by keeping only a subset of patches visible. Additionally, considering there may be different patches with similar visual appearances, we propose position smoothing and attentive reconstruction strategies to relax this classification problem, since it is not necessary to reconstruct their exact positions in these cases. Empirical evaluations of DropPos show strong capabilities. DropPos outperforms supervised pre-training and achieves competitive results compared with state-of-the-art self-supervised alternatives on a wide range of downstream benchmarks. This suggests that explicitly encouraging spatial reasoning abilities, as DropPos does, indeed contributes to the improved location awareness of ViTs. The code is publicly available at https://github.com/Haochen-Wang409/DropPos.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。