基于跨模态GAN反演的多模态引导图像风格迁移
Multimodality-guided Image Style Transfer using Cross-modal GAN Inversion
AI总结：
本文提出跨模态GAN反演方法，将图像和文本等多模态风格输入统一为风格表示，实现多模态引导图像风格迁移，并在TIST和MMIST任务上取得最先进性能。
摘要：
图像风格迁移（IST）是计算机视觉与艺术的交叉课题，持续吸引着研究者的兴趣。与传统的以风格参考图像作为输入来定义所需风格的图像引导图像风格迁移（IIST）方法不同，近期工作开始以文本引导的方式解决该问题，即文本引导图像风格迁移（TIST）。与IIST相比，这类方法通过文本指定风格提供了更大的灵活性，在难以用参考图像定义风格的场景中尤为有用。遗憾的是，许多TIST方法在迁移后的图像中会产生不理想的伪影。为解决这一问题，我们提出了一种新方法，在文本引导的基础上实现了显著改进的风格迁移。同时，为了提供比IIST和TIST更大的灵活性，我们的方法允许来自多个来源和模态的风格输入，从而实现多模态引导图像风格迁移（MMIST）。具体而言，我们通过一种新颖的跨模态GAN反演方法实现MMIST，该方法生成与指定风格一致的风格表示。这种风格表示有助于风格迁移，并且原则上可将任何IIST方法推广到MMIST。大规模实验和用户研究表明，我们的方法在TIST任务上取得了最先进的性能。此外，全面的定性结果证实了我们的方法在MMIST任务和跨模态风格插值上的有效性。
英文摘要：
Image Style Transfer (IST) is an interdisciplinary topic of computer vision and art that continuously attracts researchers' interests. Different from traditional Image-guided Image Style Transfer (IIST) methods that require a style reference image as input to define the desired style, recent works start to tackle the problem in a text-guided manner, i.e., Text-guided Image Style Transfer (TIST). Compared to IIST, such approaches provide more flexibility with text-specified styles, which are useful in scenarios where the style is hard to define with reference images. Unfortunately, many TIST approaches produce undesirable artifacts in the transferred images. To address this issue, we present a novel method to achieve much improved style transfer based on text guidance. Meanwhile, to offer more flexibility than IIST and TIST, our method allows style inputs from multiple sources and modalities, enabling MultiModality-guided Image Style Transfer (MMIST). Specifically, we realize MMIST with a novel cross-modal GAN inversion method, which generates style representations consistent with specified styles. Such style representations facilitate style transfer and in principle generalize any IIST methods to MMIST. Large-scale experiments and user studies demonstrate that our method achieves state-of-the-art performance on TIST task. Furthermore, comprehensive qualitative results confirm the effectiveness of our method on MMIST task and cross-modal style interpolation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
