arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.06304cs.CVcs.AIcs.LG
收藏

Patch n' Pack:NaViT,一种适用于任意宽高比和分辨率的 Vision Transformer

Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution

Google DeepMind(谷歌DeepMind)
Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim Ala… 展开作者

Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim Alabdulmohsin, Avital Oliver, Piotr Padlewski, Alexey Gritsenko, Mario Lučić, Neil Houlsby

AI总结:

本文提出 NaViT,通过训练时序列打包支持任意分辨率与宽高比,提升预训练效率、下游迁移表现及测试时成本-性能权衡灵活性。

摘要:

在使用计算机视觉模型处理图像之前,将图像调整为固定分辨率是一种普遍存在且已被证明次优的选择,但这一做法尚未得到成功挑战。然而,Vision Transformer(ViT)等模型提供了灵活的基于序列的建模,因此可支持变化的输入序列长度。我们利用这一点提出了 NaViT(Native Resolution ViT),它在训练期间使用序列打包来处理任意分辨率和宽高比的输入。除了灵活的模型使用方式外,我们还证明了其在大规模监督式和对比式图像-文本预训练中的训练效率得到提升。NaViT 可以高效迁移到图像与视频分类、目标检测和语义分割等标准任务,并在鲁棒性和公平性基准上带来改进结果。在推理时,输入分辨率的灵活性可用于平滑地权衡测试时的成本与性能。我们认为,NaViT 标志着对大多数计算机视觉模型所采用的、为 CNN 设计的标准输入与建模流程的背离,并代表了 ViT 的一个有前景方向。

英文摘要:

The ubiquitous and demonstrably suboptimal choice of resizing images to a fixed resolution before processing them with computer vision models has not yet been successfully challenged. However, models such as the Vision Transformer (ViT) offer flexible sequence-based modeling, and hence varying input sequence lengths. We take advantage of this with NaViT (Native Resolution ViT) which uses sequence packing during training to process inputs of arbitrary resolutions and aspect ratios. Alongside flexible model usage, we demonstrate improved training efficiency for large-scale supervised and contrastive image-text pretraining. NaViT can be efficiently transferred to standard tasks such as image and video classification, object detection, and semantic segmentation and leads to improved results on robustness and fairness benchmarks. At inference time, the input resolution flexibility can be used to smoothly navigate the test-time cost-performance trade-off. We believe that NaViT marks a departure from the standard, CNN-designed, input and modelling pipeline used by most computer vision models, and represents a promising direction for ViTs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑