语义图像合成中的类自适应交叉注意力
Semantic Image Synthesis via Class-Adaptive Cross-Attention
AI总结：
针对SPADE层忽略全局统计和限制形状操作的问题，提出用交叉注意力替代SPADE，学习形状-风格相关性，实现更优的生成质量和风格迁移。
摘要：
在语义图像合成领域，最先进的方法主要采用SPatially-Adaptive DE-normalization（SPADE）层的定制变体，这些层能够实现良好的视觉生成质量和编辑灵活性。设计上，此类层学习逐像素的调制参数，根据每个像素所属的语义类别对生成器激活进行去归一化。因此，它们往往忽略全局图像统计信息，最终导致局部风格编辑不令人信服，并引发全局不一致性，如颜色或光照分布偏移。此外，SPADE层需要语义分割掩码来映射生成器中的风格，从而阻碍了无需手动干预的形状操作。为此，我们设计了一种新颖架构，用交叉注意力层替代SPADE，以学习形状-风格相关性，从而调节图像生成过程。我们的模型继承了SPADE的灵活性，同时获得了最先进的生成质量，以及改进的全局和局部风格迁移。代码和模型可在https://github.com/TFonta/CA2SIS获取。
英文摘要：
In semantic image synthesis the state of the art is dominated by methods that use customized variants of the SPatially-Adaptive DE-normalization (SPADE) layers, which allow for good visual generation quality and editing versatility. By design, such layers learn pixel-wise modulation parameters to de-normalize the generator activations based on the semantic class each pixel belongs to. Thus, they tend to overlook global image statistics, ultimately leading to unconvincing local style editing and causing global inconsistencies such as color or illumination distribution shifts. Also, SPADE layers require the semantic segmentation mask for mapping styles in the generator, preventing shape manipulations without manual intervention. In response, we designed a novel architecture where cross-attention layers are used in place of SPADE for learning shape-style correlations and so conditioning the image generation process. Our model inherits the versatility of SPADE, at the same time obtaining state-of-the-art generation quality, as well as improved global and local style transfer. Code and models available at https://github.com/TFonta/CA2SIS.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
