Diffusion-based text-to-image generation models like GLIDE and DALLE-2 have gained wide success recently for their superior performance in turning complex text inputs into images of high quality and wide diversity. In particular, they are proven to be very powerful in creating graphic arts of various formats and styles. Although current models supported specifying style formats like oil painting or pencil drawing, fine-grained style features like color distributions and brush strokes are hard to specify as they are randomly picked from a conditional distribution based on the given text input. Here we propose a novel style guidance method to support generating images using arbitrary style guided by a reference image. The generation method does not require a separate style transfer model to generate desired styles while maintaining image quality in generated content as controlled by the text input. Additionally, the guidance method can be applied without a style reference, denoted as self style guidance, to generate images of more diverse styles. Comprehensive experiments prove that the proposed method remains robust and effective in a wide range of conditions, including diverse graphic art forms, image content types and diffusion models.
Imperial College London（伦敦帝国学院）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希·亚历山大大学）
This paper presents the use of Multi-Agent Reinforcement Learning (MARL) to perform navigation in 3D anatomical volumes from medical imaging. We utilize Neural Style Transfer to create synthetic Computed Tomography (CT) agent gym environments and assess the generalization capabilities of our agents to clinical CT volumes. Our framework does not require any labelled clinical data and integrates easily with several image translation techniques, enabling cross modality applications. Further, we solely condition our agents on 2D slices, breaking grounds for 3D guidance in much more difficult imagi…
TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；College of Computer Science, Nankai University（南开大学计算机学院）
In recent years, arbitrary image style transfer has attracted more and more attention. Given a pair of content and style images, a stylized one is hoped that retains the content from the former while catching style patterns from the latter. However, it is difficult to simultaneously keep well the trade-off between the content details and the style features. To stylize the image with sufficient style patterns, the content details may be damaged and sometimes the objects of images can not be distinguished clearly. For this reason, we present a new transformer-based method named STT for image sty…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Microsoft Research Asia（微软亚洲研究院）；Kuaishou Technology（快手科技）
Multimodal and multi-domain stylization are two important problems in the field of image style transfer. Currently, there are few methods that can perform both multimodal and multi-domain stylization simultaneously. In this paper, we propose a unified framework for multimodal and multi-domain style transfer with the support of both exemplar-based reference and randomly sampled guidance. The key component of our method is a novel style distribution alignment module that eliminates the explicit distribution gaps between various style domains and reduces the risk of mode collapse. The multimodal …
Dual Orthogonal Guidance for Robust Diffusion-based Handwritten Text Generation
Luleå University of Technology（卢莱大学）；National Technical University of Athens（雅典技术大学）；University of West Attica（西阿提卡大学）；University of Modena and Reggio Emilia（摩德纳和雷吉奥艾米利亚大学）
Tune-Your-Style: Intensity-tunable 3D Style Transfer with Gaussian Splatting
School of Electronic and Computer Engineering, Peking University, Shenzhen, China（电子与计算机工程学院，北京大学深圳校区）；Pengcheng Laboratory, Shenzhen, China（鹏城实验室）；AI for Science (AI4S)-Preferred Program, Peking University Shenzhen Graduate School, China（人工智能科学（AI4S）优选计划，北京大学深圳研究生院）；Department of Automation and BNRist, Tsinghua University, Beijing, China（自动化系和BNRist，清华大学，北京，中国）；Dalian University of Technology, China（大连理工大学）
Recently, with the revolutionary neural style transferring methods, creditable paintings can be synthesized automatically from content images and style images. However, when it comes to the task of applying a painting's style to an anime sketch, these methods will just randomly colorize sketch lines as outputs and fail in the main task: specific style tranfer. In this paper, we integrated residual U-net to apply the style to the gray-scale sketch with auxiliary classifier generative adversarial network (AC-GAN). The whole process is automatic and fast, and the results are creditable in the qua…
OMGTex: One-stage Multi-style Facial Texture Reconstruction without Geometry Guidance
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）科学与工程学院）；Guangdong Provincial Key Laboratory of Future Networks of Intelligence（广东省未来网络智能化重点实验室）；FNii-Shenzhen（FNii-深圳）
Through Van Gogh's Eyes: Global Style Transfer with Diffusion Model
Korea Institute of Science and Technology（韩国科学技术研究院）；University of Science and Technology（科技大学）；Korea University（高丽大学）；Gachon University（嘉泉大学）；Kyung Hee University（庆熙大学）
Motion Transfer-Enhanced StyleGAN for Generating Diverse Macaque Facial Expressions
Human Informatics Research Institute, National Institute of Advanced Industrial Science and Technology (AIST)（国家先进工业科学与技术研究所（AIST）人类信息学研究所）；Graduate School of Engineering, The University of Tokyo（东京大学工学研究科）；Human Biology & Primate Cognition, Institute of Biology, Leipzig University（莱比锡大学生物研究所人类生物学与灵长类认知）；Araya Inc.（Araya公司）；Center for the Evolutionary Origins of Human Behavior (EHuB), Kyoto University（京都大学人类行为进化起源中心（EHuB））
Neural style transfer has been demonstrated to be powerful in creating artistic image with help of Convolutional Neural Networks (CNN). However, there is still lack of computational analysis of perceptual components of the artistic style. Different from some early attempts which studied the style by some pre-processing or post-processing techniques, we investigate the characteristics of the style systematically based on feature map produced by CNN. First, we computationally decompose the style into basic elements using not only spectrum based methods including Fast Fourier Transform (FFT), Dis…
School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））
Recent studies have shown remarkable success in universal style transfer which transfers arbitrary visual styles to content images. However, existing approaches suffer from the aesthetic-unrealistic problem that introduces disharmonious patterns and evident artifacts, making the results easy to spot from real paintings. To address this limitation, we propose AesUST, a novel Aesthetic-enhanced Universal Style Transfer approach that can generate aesthetically more realistic and pleasing results for arbitrary styles. Specifically, our approach introduces an aesthetic discriminator to learn the un…
Inception Institute of Artificial Intelligence（起源人工智能研究院）；EPFL（洛桑联邦理工学院）；Lausanne University Hospital（洛桑大学医院）；Center of Biomedical Imaging（生物医学成像中心）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Although generative adversarial network (GAN) based style transfer is state of the art in histopathology color-stain normalization, they do not explicitly integrate structural information of tissues. We propose a self-supervised approach to incorporate semantic guidance into a GAN based stain normalization framework and preserve detailed structural information. Our method does not require manual segmentation maps which is a significant advantage over existing methods. We integrate semantic information at different layers between a pre-trained semantic network and the stain color normalization …
UC Berkeley（加州大学伯克利分校）；Picsart AI Research (PAIR)（Picsart人工智能研究院（PAIR））；University of Oregon（俄勒冈大学）；The University of Hong Kong（香港大学）
Controllable image synthesis models allow creation of diverse images based on text instructions or guidance from a reference image. Recently, denoising diffusion probabilistic models have been shown to generate more realistic imagery than prior methods, and have been successfully demonstrated in unconditional and class-conditional settings. We investigate fine-grained, continuous control of this model class, and introduce a novel unified framework for semantic diffusion guidance, which allows either language or image guidance, or both. Guidance is injected into a pretrained unconditional diffu…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Jilin University（吉林大学）；Kuaishou Technology（快手科技）；National Cheng-Kung University（国立成功大学）
In this work, we tackle the challenging problem of arbitrary image style transfer using a novel style feature representation learning method. A suitable style representation, as a key component in image stylization tasks, is essential to achieve satisfactory results. Existing deep neural network based approaches achieve reasonable results with the guidance from second-order statistics such as Gram matrix of content features. However, they do not leverage sufficient style information, which results in artifacts such as local distortions and style inconsistency. To address these issues, we propo…
Generating realistic motions for digital humans is a core but challenging part of computer animations and games, as human motions are both diverse in content and rich in styles. While the latest deep learning approaches have made significant advancements in this domain, they mostly consider motion synthesis and style manipulation as two separate problems. This is mainly due to the challenge of learning both motion contents that account for the inter-class behaviour and styles that account for the intra-class behaviour effectively in a common representation. To tackle this challenge, we propose…
Toyota Technological Institute at Chicago（芝加哥丰田理工学院）；University of Chicago（芝加哥大学）
Style transfer algorithms strive to render the content of one image using the style of another. We propose Style Transfer by Relaxed Optimal Transport and Self-Similarity (STROTSS), a new optimization-based style transfer algorithm. We extend our method to allow user-specified point-to-point or region-to-region control over visual similarity between the style image and the output. Such guidance can be used to either achieve a particular visual effect or correct errors made by unconstrained style transfer. In order to quantitatively compare our method to prior work, we conduct a large-scale use…
STAY Diffusion: Styled Layout Diffusion Model for Diverse Layout-to-Image Generation
Bosch Center for Artificial Intelligence（博世人工智能中心）；Institute of Industrial Manufacturing and Management IFF, University of Stuttgart（斯图加特大学工业制造与管理研究所）；Fraunhofer Institute for Manufacturing Engineering and Automation IPA（弗劳恩霍夫制造工程与自动化研究所）
针对布局到图像生成，本文提出 STAY Diffusion，通过 EA Norm、自监督语义图与 SM Attention 增强全局条件和对象关系建模，提升图像质量、多样性、准确性与可控性。
Boston University（波士顿大学）；Google Research（谷歌研究院）；Google（谷歌）
We present a novel algorithm for transferring artistic styles of semantically meaningful local regions of an image onto local regions of a target video while preserving its photorealism. Local regions may be selected either fully automatically from an image, through using video segmentation algorithms, or from casual user guidance such as scribbles. Our method, based on a deep neural network architecture inspired by recent work in photorealistic style transfer, is real-time and works on arbitrary inputs without runtime optimization once trained on a diverse dataset of artistic styles. By augme…
StyleStudio: Text-Driven Style Transfer with Selective Control of Style Elements
Westlake University（西湖大学）；Fudan University（复旦大学）；Nanyang Technological University（南洋理工大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
HiCAST: Highly Customized Arbitrary Style Transfer with Adapter Enhanced Diffusion Models
Harbin Institute of Technology（哈尔滨工业大学）；Baidu Research（百度研究院）；Baidu Inc.（百度公司）；University of Chinese Academy of Sciences（中国科学院大学）；ETH Zurich（苏黎世联邦理工学院）
Evaluation of Randomization through Style Transfer for Enhanced Domain Generalization
German Cancer Research Center (DKFZ)（德国癌症研究中心（DKFZ））；German Cancer Consortium (DKTK), partner site Frankfurt/Mainz, a partnership between DKFZ and UCT Frankfurt-Marburg（德国癌症联盟（DKTK），法兰克福/美因茨合作站点，由DKFZ与UCT法兰克福-马尔堡合作）；Goethe University Frankfurt, Department of Computer Science（法兰克福大学计算机科学系）；The Hessian Center for Artificial Intelligence (hessian.AI)（黑森人工智能中心（hessian.AI））
Training-free Subject-Enhanced Attention Guidance for Compositional Text-to-image Generation
University of Chinese Academy of Sciences, Beijing, China（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所）；Kuaishou Technology, Beijing, China（快手科技）
