arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.07550cs.CV
收藏

一张图像在重建与生成中仅值32个Token

An Image is Worth 32 Tokens for Reconstruction and Generation

Technical University Munich(慕尼黑工业大学);ByteDance(字节跳动)

Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, Liang-Chieh Chen

AI总结:

本文提出TiTok一维图像分词器,将图像压缩为32个离散token,在ImageNet重建与生成基准上以更少token和更快速度达到或超过MaskGIT与DiT-XL/2。

摘要:

生成模型的最新进展凸显了图像分词在高效合成高分辨率图像中的关键作用。分词将图像转换为潜在表示,与直接处理像素相比降低了计算需求,并提升了生成过程的效果和效率。以往方法(如VQGAN)通常使用具有固定下采样因子的二维潜在网格。然而,这些二维分词方法在处理图像中固有的冗余时面临挑战,即相邻区域经常表现出相似性。为解决这一问题,我们提出了基于Transformer的一维分词器(TiTok),这是一种将图像分词为一维潜在序列的创新方法。TiTok提供了更紧凑的潜在表示,相比传统技术产生了显著更高效、更有效的表示。例如,一张256×256×3的图像可被压缩为仅32个离散token,而以往方法需要256或1024个token,压缩幅度显著。尽管性质紧凑,TiTok仍取得了与最先进方法相竞争的性能。具体而言,在使用相同生成器框架的情况下,TiTok达到1.97 gFID,在ImageNet 256×256基准上比MaskGIT基线显著高出4.21。当涉及更高分辨率时,TiTok的优势变得更加明显。在ImageNet 512×512基准上,TiTok不仅优于最先进的扩散模型DiT-XL/2(gFID为2.74对3.04),还将图像token减少64倍,使生成过程加快410倍。我们性能最佳的变体可显著超过DiT-XL/2(gFID为2.13对3.04),同时仍能以74倍的速度生成高质量样本。

英文摘要:

Recent advancements in generative models have highlighted the crucial role of image tokenization in the efficient synthesis of high-resolution images. Tokenization, which transforms images into latent representations, reduces computational demands compared to directly processing pixels and enhances the effectiveness and efficiency of the generation process. Prior methods, such as VQGAN, typically utilize 2D latent grids with fixed downsampling factors. However, these 2D tokenizations face challenges in managing the inherent redundancies present in images, where adjacent regions frequently display similarities. To overcome this issue, we introduce Transformer-based 1-Dimensional Tokenizer (TiTok), an innovative approach that tokenizes images into 1D latent sequences. TiTok provides a more compact latent representation, yielding substantially more efficient and effective representations than conventional techniques. For example, a 256 x 256 x 3 image can be reduced to just 32 discrete tokens, a significant reduction from the 256 or 1024 tokens obtained by prior methods. Despite its compact nature, TiTok achieves competitive performance to state-of-the-art approaches. Specifically, using the same generator framework, TiTok attains 1.97 gFID, outperforming MaskGIT baseline significantly by 4.21 at ImageNet 256 x 256 benchmark. The advantages of TiTok become even more significant when it comes to higher resolution. At ImageNet 512 x 512 benchmark, TiTok not only outperforms state-of-the-art diffusion model DiT-XL/2 (gFID 2.74 vs. 3.04), but also reduces the image tokens by 64x, leading to 410x faster generation process. Our best-performing variant can significantly surpasses DiT-XL/2 (gFID 2.13 vs. 3.04) while still generating high-quality samples 74x faster.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑