让图像变换器对齐表示并非如你所想的那么简单
Representation Alignment for Just Image Transformers is not Easier than You Think
AI总结：
本文指出，对于Just Image Transformers（JiT）而言，传统的REPA方法在训练过程中表现不佳，导致FID值升高和多样性下降。作者提出PixelREPA方法，通过调整对齐目标并结合掩码变换器适配器，提升了训练收敛性和最终质量。
摘要：
表示对齐（REPA）已成为加速扩散变换器在潜在空间训练的一种简单方法。同时，像Just Image Transformers（JiT）这样的像素空间扩散变换器由于不依赖预训练的分词器，从而避免了潜在扩散的重建瓶颈，而受到越来越多的关注。本文表明，REPA在JiT上会失效。REPA在训练过程中导致JiT的FID值变差，并在预训练语义编码器的ImageNet表示空间中，对紧密聚类的图像子集造成多样性崩溃。我们追溯失败的原因在于信息不对称：去噪发生在高维图像空间中，而语义目标被强烈压缩，使直接回归成为捷径目标。我们提出PixelREPA，它将对齐目标转换并结合一个结合浅层变换器适配器和部分标记掩码的Masked Transformer Adapter进行约束。PixelREPA在训练收敛性和最终质量上都有所提升。PixelREPA将JiT-B$/16$的FID从3.66降低到3.17，并在ImageNet $256 \times 256$上将Inception Score（IS）从275.1提升到284.6，同时实现了> 2倍的更快收敛。最后，PixelREPA-H$/16$实现了FID=1.81和IS=317.2。我们的代码可在https://github.com/kaist-cvml/PixelREPA上获得。
英文摘要：
Representation Alignment (REPA) has emerged as a simple way to accelerate Diffusion Transformers training in latent space. At the same time, pixel-space diffusion transformers such as Just image Transformers (JiT) have attracted growing attention because they remove a dependency on a pretrained tokenizer, and then avoid the reconstruction bottleneck of latent diffusion. This paper shows that the REPA can fail for JiT. REPA yields worse FID for JiT as training proceeds and collapses diversity on image subsets that are tightly clustered in the representation space of pretrained semantic encoder on ImageNet. We trace the failure to an information asymmetry: denoising occurs in the high dimensional image space, while the semantic target is strongly compressed, making direct regression a shortcut objective. We propose PixelREPA, which transforms the alignment target and constrains alignment with a Masked Transformer Adapter that combines a shallow transformer adapter with partial token masking. PixelREPA improves both training convergence and final quality. PixelREPA reduces FID from 3.66 to 3.17 for JiT-B$/16$ and improves Inception Score (IS) from 275.1 to 284.6 on ImageNet $256 \times 256$, while achieving $> 2\times$ faster convergence. Finally, PixelREPA-H$/16$ achieves FID$=1.81$ and IS$=317.2$. Our code is available at https://github.com/kaist-cvml/PixelREPA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
