arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.12376cs.CV
收藏

FiT:用于扩散模型的灵活视觉Transformer

FiT: Flexible Vision Transformer for Diffusion Model

Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Sydney University(悉尼大学);The University of Hong Kong(香港大学)

Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, Lei Bai

AI总结:

本文提出Flexible Vision Transformer (FiT),通过将图像概念化为动态大小token序列,使扩散模型能灵活生成不受限分辨率和长宽比的图像,并消除裁剪偏差,在分辨率泛化和外推生成上表现优异。

摘要:

自然本质上是无限分辨率的。在这一现实背景下,现有的扩散模型(如Diffusion Transformers)在处理其训练域外的图像分辨率时常常面临挑战。为了克服这一限制,我们提出了Flexible Vision Transformer (FiT),这是一种专门设计用于生成不受限制分辨率和长宽比图像的transformer架构。与传统方法将图像视为静态分辨率网格不同,FiT将图像概念化为动态大小token的序列。这一视角实现了灵活的训练策略,在训练和推理阶段都能轻松适应各种长宽比,从而促进了分辨率泛化并消除了由图像裁剪引起的偏差。通过精心调整的网络结构和无需训练的外推技术的整合,FiT在分辨率外推生成方面展现出卓越的灵活性。综合实验表明,FiT在广泛的分辨率范围内表现出色,展示了其在训练分辨率分布内外的有效性。代码库可在https://github.com/whlzy/FiT获取。

英文摘要:

Nature is infinitely resolution-free. In the context of this reality, existing diffusion models, such as Diffusion Transformers, often face challenges when processing image resolutions outside of their trained domain. To overcome this limitation, we present the Flexible Vision Transformer (FiT), a transformer architecture specifically designed for generating images with unrestricted resolutions and aspect ratios. Unlike traditional methods that perceive images as static-resolution grids, FiT conceptualizes images as sequences of dynamically-sized tokens. This perspective enables a flexible training strategy that effortlessly adapts to diverse aspect ratios during both training and inference phases, thus promoting resolution generalization and eliminating biases induced by image cropping. Enhanced by a meticulously adjusted network structure and the integration of training-free extrapolation techniques, FiT exhibits remarkable flexibility in resolution extrapolation generation. Comprehensive experiments demonstrate the exceptional performance of FiT across a broad range of resolutions, showcasing its effectiveness both within and beyond its training resolution distribution. Repository available at https://github.com/whlzy/FiT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑