arXiv 2211.153132022-11-28cs.CVcs.AI

MicroAST：超高速超分辨率任意格式传输

MicroAST: Towards Super-Fast Ultra-Resolution Arbitrary Style Transfer

College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）

Zhizhong Wang, Lei Zhao, Zhiwen Zuo, Ailin Li, Haibo Chen, Wei Xing, Dongming Lu

备注：: Accepted by AAAI 2023

摘要：

任意风格转移（AST）将任意艺术风格转移到内容图像上。尽管最近进展迅速，但现有的AST方法要么不能要么太慢而不能以超分辨率运行（例如，4K），资源有限，这严重阻碍了它们的进一步应用。在本文中，我们通过学习一个简单而轻量级的模型来解决这个难题，称为MicroAST。关键的见解是完全放弃使用繁琐的预先训练的深度卷积神经网络（例如，VGG）在推理时。相反，我们设计了两个微编码器（内容和风格编码器）和一个微解码器的风格转移。内容编码器的目标是提取内容图像的主要结构。与调制器耦合的风格编码器将风格图像编码成可学习的双调制信号，该双调制信号调制解码器的中间特征和卷积滤波器两者，从而注入更复杂和灵活的风格信号来引导风格化。此外，为了提高风格编码器提取更清晰、更有代表性的风格信号的能力，我们还在模型中引入了一种新的风格信号对比度损失。与最先进的技术相比，我们的MicroAST不仅能产生视觉上优越的结果，而且体积小5-73倍，速度快6-18倍，首次实现了4K超分辨率下的超快（约0.5秒）AST。代码可在https://github.com/EndyWon/MicroAST上获得。

英文摘要：

Arbitrary style transfer (AST) transfers arbitrary artistic styles onto content images. Despite the recent rapid progress, existing AST methods are either incapable or too slow to run at ultra-resolutions (e.g., 4K) with limited resources, which heavily hinders their further applications. In this paper, we tackle this dilemma by learning a straightforward and lightweight model, dubbed MicroAST. The key insight is to completely abandon the use of cumbersome pre-trained Deep Convolutional Neural Networks (e.g., VGG) at inference. Instead, we design two micro encoders (content and style encoders) and one micro decoder for style transfer. The content encoder aims at extracting the main structure of the content image. The style encoder, coupled with a modulator, encodes the style image into learnable dual-modulation signals that modulate both intermediate features and convolutional filters of the decoder, thus injecting more sophisticated and flexible style signals to guide the stylizations. In addition, to boost the ability of the style encoder to extract more distinct and representative style signals, we also introduce a new style signal contrastive loss in our model. Compared to the state of the art, our MicroAST not only produces visually superior results but also is 5-73 times smaller and 6-18 times faster, for the first time enabling super-fast (about 0.5 seconds) AST at 4K ultra-resolutions. Code is available at https://github.com/EndyWon/MicroAST.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文