SwinStyleformer是图像逆映射的优选方案
SwinStyleformer is a favorable choice for image inversion
AI总结：
针对Transformer骨干网络难以完成图像逆映射的问题，本文提出纯Transformer结构的SwinStyleformer，通过小窗口骨干、可学习查询模块、多尺度连接及分布对齐等设计，实现图像逆映射的SOTA性能。
摘要：
本文提出了首个纯Transformer结构的逆映射网络SwinStyleformer，该网络能够通过处理长距离依赖、学习目标的全局结构，弥补CNN逆映射框架的不足。实验发现，采用Transformer骨干网络的逆映射网络无法成功完成图像逆映射。上述现象源于CNN与Transformer之间的差异：例如与卷积相比，自注意力权重更侧重图像结构而忽略图像细节、Transformer缺乏多尺度特性，以及Transformer提取的隐编码与StyleGAN风格向量之间存在分布差异。为解决这些差异，我们采用窗口尺寸更小的Swin Transformer作为SwinStyleformer的骨干网络，以增强逆映射图像的局部细节。同时，我们设计了一种基于可学习查询的Transformer模块。与自注意力Transformer模块相比，基于可学习查询的Transformer模块具备更强的适应性与灵活性，能够让模型根据特定任务更新注意力权重，从而使逆映射的关注点不再局限于图像结构。为进一步引入多尺度特性，我们在特征图提取过程中设计了多尺度连接。多尺度连接可让模型全面理解图像，避免因全局建模导致的细节丢失。此外，我们提出了逆映射判别器与分布对齐损失，以最小化分布差异。基于上述设计，我们的SwinStyleformer成功解决了Transformer逆映射失败的问题，并在图像逆映射及多项相关视觉任务中展现出SOTA（当前最优）性能。
英文摘要：
This paper proposes the first pure Transformer structure inversion network called SwinStyleformer, which can compensate for the shortcomings of the CNNs inversion framework by handling long-range dependencies and learning the global structure of objects. Experiments found that the inversion network with the Transformer backbone could not successfully invert the image. The above phenomena arise from the differences between CNNs and Transformers, such as the self-attention weights favoring image structure ignoring image details compared to convolution, the lack of multi-scale properties of Transformer, and the distribution differences between the latent code extracted by the Transformer and the StyleGAN style vector. To address these differences, we employ the Swin Transformer with a smaller window size as the backbone of the SwinStyleformer to enhance the local detail of the inversion image. Meanwhile, we design a Transformer block based on learnable queries. Compared to the self-attention transformer block, the Transformer block based on learnable queries provides greater adaptability and flexibility, enabling the model to update the attention weights according to specific tasks. Thus, the inversion focus is not limited to the image structure. To further introduce multi-scale properties, we design multi-scale connections in the extraction of feature maps. Multi-scale connections allow the model to gain a comprehensive understanding of the image to avoid loss of detail due to global modeling. Moreover, we propose an inversion discriminator and distribution alignment loss to minimize the distribution differences. Based on the above designs, our SwinStyleformer successfully solves the Transformer's inversion failure issue and demonstrates SOTA performance in image inversion and several related vision tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
