基于多模态引导GAN逆变换的高保真图像修复
High-Fidelity Image Inpainting with Multimodal Guided GAN Inversion
AI总结：
针对现有GAN逆变换图像修复方法忽略未掩码区域一致性约束及单模态限制的问题，提出MMInvertFill，通过多模态引导编码器、F&W+潜在空间及软更新均值潜在模块，在六个数据集上超越现有方法，有效支持域外图像补全。
摘要：
生成对抗网络（GAN）逆变换在图像修复任务中展现出卓越性能，该任务旨在利用未掩码内容恢复丢失或损坏的图像纹理。以往基于GAN逆变换的方法通常将训练良好的GAN模型作为有效先验，为缺失区域生成真实内容。尽管性能出色，但它们忽略了一个硬性约束：输入和输出中的未掩码区域应保持一致，这导致GAN逆变换与图像修复之间存在差距，从而降低性能。此外，现有GAN逆变换方法常仅考虑输入图像的单一模态，忽略了图像中的其他辅助线索以提升性能。针对这些问题，我们提出一种名为MMInvertFill的新型GAN逆变换方法用于图像修复。MMInvertFill主要包含一个带预调制的多模态引导编码器和一个具有F&W+潜在空间的GAN生成器。具体而言，多模态编码器旨在通过门控掩码感知注意力模块，结合额外的语义分割边缘纹理模态增强多尺度结构。随后，提出预调制将这些结构编码为风格向量。为缓解明显的颜色差异和语义不一致问题，我们引入F&W+潜在空间以弥合GAN逆变换与图像修复之间的差距。此外，为重建忠实且逼真的图像，我们设计了一个简单但有效的软更新均值潜在模块，以捕捉更多样化的域内模式，为大规模损坏生成高保真纹理。在六个具有挑战性的数据集上进行的大量实验表明，我们的MMInvertFill在定性和定量上均优于其他最先进方法，且能有效支持域外图像的补全。
英文摘要：
Generative Adversarial Network (GAN) inversion have demonstrated excellent performance in image inpainting that aims to restore lost or damaged image texture using its unmasked content. Previous GAN inversion-based methods usually utilize well-trained GAN models as effective priors to generate the realistic regions for missing holes. Despite excellence, they ignore a hard constraint that the unmasked regions in the input and the output should be the same, resulting in a gap between GAN inversion and image inpainting and thus degrading the performance. Besides, existing GAN inversion approaches often consider a single modality of the input image, neglecting other auxiliary cues in images for improvements. Addressing these problems, we propose a novel GAN inversion approach, dubbed MMInvertFill, for image inpainting. MMInvertFill contains primarily a multimodal guided encoder with a pre-modulation and a GAN generator with F&W+ latent space. Specifically, the multimodal encoder aims to enhance the multi-scale structures with additional semantic segmentation edge texture modalities through a gated mask-aware attention module. Afterwards, a pre-modulation is presented to encode these structures into style vectors. To mitigate issues of conspicuous color discrepancy and semantic inconsistency, we introduce the F&W+ latent space to bridge the gap between GAN inversion and image inpainting. Furthermore, in order to reconstruct faithful and photorealistic images, we devise a simple yet effective Soft-update Mean Latent module to capture more diversified in-domain patterns for generating high-fidelity textures for massive corruptions. In our extensive experiments on six challenging datasets, we show that our MMInvertFill qualitatively and quantitatively outperforms other state-of-the-arts and it supports the completion of out-of-domain images effectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
