arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2208.06140cs.CVcs.LGeess.IV
收藏

风格分光镜:通过傅立叶分析提高解释性和可控性

Style Spectroscope: Improve Interpretability and Controllability through Fourier Analysis

Fudan University(复旦大学)

Zhiyu Jin, Xuli Shen, Bin Li, Xiangyang Xue

摘要:

通用样式传输(UST)将来自任意参考图像的样式注入到内容图像中。现有的方法,虽然享有许多实际的成功,是无法解释的实验观察,包括不同的性能UST算法在保持内容图像的空间结构。此外,方法仅限于繁琐的全局样式化控制,因此它们需要额外的空间遮罩来实现所需的样式化。在这项工作中,我们提供了一个系统的傅立叶分析UST的一般框架。我们提出了一个等效形式的框架在频域中。该形式意味着现有算法平等地对待特征图的所有频率分量和像素,除了零频率分量。我们将傅立叶振幅和相位分别与Gram矩阵和风格转换中的内容重建损失相联系。基于这种等价性和联系,我们可以解释具有傅立叶相位的算法之间的不同结构保持行为。鉴于我们的解释,我们提出了两个操作在实践中的结构保护和所需的风格化。定性和定量实验都证明了我们的方法对国家的最先进的方法的竞争性能。我们还进行实验来证明(1)上述等价性,(2)基于傅立叶振幅和相位的可解释性和(3)与频率分量相关的可控性。

英文摘要:

Universal style transfer (UST) infuses styles from arbitrary reference images into content images. Existing methods, while enjoying many practical successes, are unable of explaining experimental observations, including different performances of UST algorithms in preserving the spatial structure of content images. In addition, methods are limited to cumbersome global controls on stylization, so that they require additional spatial masks for desired stylization. In this work, we provide a systematic Fourier analysis on a general framework for UST. We present an equivalent form of the framework in the frequency domain. The form implies that existing algorithms treat all frequency components and pixels of feature maps equally, except for the zero-frequency component. We connect Fourier amplitude and phase with Gram matrices and a content reconstruction loss in style transfer, respectively. Based on such equivalence and connections, we can thus interpret different structure preservation behaviors between algorithms with Fourier phase. Given the interpretations we have, we propose two manipulations in practice for structure preservation and desired stylization. Both qualitative and quantitative experiments demonstrate the competitive performance of our method against the state-of-the-art methods. We also conduct experiments to demonstrate (1) the abovementioned equivalence, (2) the interpretability based on Fourier amplitude and phase and (3) the controllability associated with frequency components.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑