从视觉基础模型自举 SparseFormer
Bootstrapping SparseFormers from Vision Foundation Models
AI总结：
本文提出从大规模预训练 ViT/CLIP 继承并冻结权重，仅训练轻量 focusing transformer 和少量早期层来自举 SparseFormer，以少量无标签数据实现高准确率、零样本能力及高效多模态编码。
摘要：
最近提出的 SparseFormer 架构通过调整 RoI 来使用数量显著更少的视觉 token，为视觉理解提供了一种替代方法，在大幅降低计算成本的同时仍能取得可观性能。然而，从头训练 SparseFormer 仍然代价高昂，而且扩展参数数量可能颇具挑战。本文提出以简单高效的方式从基于 ViT 的视觉基础模型自举 SparseFormer。由于大多数 SparseFormer 块都是标准 transformer 块，我们可以继承大规模预训练 vision transformer 的权重，并尽可能将其冻结。因此，我们只需训练 SparseFormer 特有的轻量级 focusing transformer 来调整 token RoI，并微调少数几个早期预训练块以对齐最终 token 表示。通过这种方式，我们可以利用相当少量的训练样本（例如 IN-1K），在没有标签或 caption 的情况下，仅用数小时便从各种大规模预训练模型（例如在 IN-21K 上预训练的 AugReg 或 CLIP）自举 SparseFormer 架构。结果表明，自 AugReg-ViT-L/16-384 自举的单模态 SparseFormer 仅使用 49 个 token 即可在 IN-1K 上达到 84.9% 的准确率；而自 CLIP 自举的多模态 SparseFormer 也在自举过程中未见过任何 caption 的情况下，以大幅降低的计算成本展现出显著的零样本性能。此外，由 CLIP 自举的 SparseFormer 在未见过任何词语的情况下将输出空间与语言对齐，可作为多模态大语言模型中的高效视觉编码器。代码和模型可在 https://github.com/showlab/sparseformer 获取。
英文摘要：
The recently proposed SparseFormer architecture provides an alternative approach to visual understanding by utilizing a significantly lower number of visual tokens via adjusting RoIs, greatly reducing computational costs while still achieving promising performance. However, training SparseFormers from scratch is still expensive, and scaling up the number of parameters can be challenging. In this paper, we propose to bootstrap SparseFormers from ViT-based vision foundation models in a simple and efficient way. Since the majority of SparseFormer blocks are the standard transformer ones, we can inherit weights from large-scale pre-trained vision transformers and freeze them as much as possible. Therefore, we only need to train the SparseFormer-specific lightweight focusing transformer to adjust token RoIs and fine-tune a few early pre-trained blocks to align the final token representation. In such a way, we can bootstrap SparseFormer architectures from various large-scale pre-trained models (e.g., IN-21K pre-trained AugRegs or CLIPs) using a rather smaller amount of training samples (e.g., IN-1K) and without labels or captions within just a few hours. As a result, the bootstrapped unimodal SparseFormer (from AugReg-ViT-L/16-384) can reach 84.9% accuracy on IN-1K with only 49 tokens, and the multimodal SparseFormer from CLIPs also demonstrates notable zero-shot performance with highly reduced computational cost without seeing any caption during the bootstrapping procedure. In addition, CLIP-bootstrapped SparseFormers, which align the output space with language without seeing a word, can serve as efficient vision encoders in multimodal large language models. Code and models are available at https://github.com/showlab/sparseformer
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
