用于语义图像合成的带多尺度对比学习的边缘引导GAN
Edge Guided GANs with Multi-Scale Contrastive Learning for Semantic Image Synthesis
AI总结：
针对语义图像合成中细节缺失、语义不一致、忽略全局语义关联三大挑战，提出带边缘引导、类别特征增强和多尺度对比学习的ECGAN，提升合成图像质量与语义一致性。
摘要：
我们针对具有挑战性的语义图像合成任务提出了一种新型ECGAN。尽管近年来该领域已取得了显著进展，但由于三大尚未解决的核心挑战，合成图像的质量仍远未达到令人满意的水平。1）语义标签无法提供详细的结构信息，导致局部细节和结构的合成难度较大；2）卷积、下采样、归一化等被广泛采用的CNN操作通常会造成空间分辨率损失，因而无法完整保留原始语义信息，进而产生语义不一致的结果（例如小目标缺失）；3）现有语义图像合成方法侧重于对单一输入语义布局中的“局部”语义信息进行建模，却忽略了多个输入语义布局的“全局”语义信息，即不同输入布局之间像素的语义交叉关联。为解决第一个挑战，我们提出将边缘作为中间表征，并通过提出的注意力引导边缘转移模块进一步利用边缘引导图像生成。为解决第二个挑战，我们设计了一个有效模块，可根据原始语义布局选择性地突出类别相关的特征图，以保留语义信息。为解决第三个挑战，受当前对比学习方法的启发，我们提出了一种新型对比学习方法，旨在迫使属于同一语义类别的像素嵌入生成比不同类别像素嵌入更相似的图像内容。我们进一步提出了一种新型多尺度对比学习方法，旨在拉近不同尺度下的同类特征的距离，通过显式挖掘不同尺度下多个输入语义布局的标记像素结构，捕获更多语义关联。
英文摘要：
We propose a novel ECGAN for the challenging semantic image synthesis task. Although considerable improvements have been achieved by the community in the recent period, the quality of synthesized images is far from satisfactory due to three largely unresolved challenges. 1) The semantic labels do not provide detailed structural information, making it challenging to synthesize local details and structures; 2) The widely adopted CNN operations such as convolution, down-sampling, and normalization usually cause spatial resolution loss and thus cannot fully preserve the original semantic information, leading to semantically inconsistent results (e.g., missing small objects); 3) Existing semantic image synthesis methods focus on modeling 'local' semantic information from a single input semantic layout. However, they ignore 'global' semantic information of multiple input semantic layouts, i.e., semantic cross-relations between pixels across different input layouts. To tackle 1), we propose to use the edge as an intermediate representation which is further adopted to guide image generation via a proposed attention guided edge transfer module. To tackle 2), we design an effective module to selectively highlight class-dependent feature maps according to the original semantic layout to preserve the semantic information. To tackle 3), inspired by current methods in contrastive learning, we propose a novel contrastive learning method, which aims to enforce pixel embeddings belonging to the same semantic class to generate more similar image content than those from different classes. We further propose a novel multi-scale contrastive learning method that aims to push same-class features from different scales closer together being able to capture more semantic relations by explicitly exploring the structures of labeled pixels from multiple input semantic layouts from different scales.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
