Controllable painting generation plays a pivotal role in image stylization. Currently, the control way of style transfer is subject to exemplar-based reference or a random one-hot vector guidance. Few works focus on decoupling the intrinsic properties of painting as control conditions, e.g., artist, genre and period. Under this circumstance, we propose a novel framework adopting multiple attributes from the painting to control the stylized results. An asymmetrical cycle structure is equipped to preserve the fidelity, associating with style preserving and attribute regression loss to keep the unique distinction of colors and textures between domains. Several qualitative and quantitative results demonstrate the effect of the combinations of multiple attributes and achieve satisfactory performance.
OPa-Ma: Text Guided Mamba for 360-degree Image Out-painting
Cleveland Clinic（克利夫兰诊所）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；East China Normal University（华东师范大学）；Solon High School（索伦高中）；Tianyuan Biotechnology（天元生物科技）
We propose a method to extrapolate a 360° field of view from a single image that allows for user-controlled synthesis of the out-painted content. To do so, we propose improvements to an existing GAN-based in-painting architecture for out-painting panoramic image representation. Our method obtains state-of-the-art results and outperforms previous methods on standard image quality metrics. To allow controlled synthesis of out-painting, we introduce a novel guided co-modulation framework, which drives the image generation process with a common pretrained discriminative model. Doing so maintains t…
Alibaba Group（阿里巴巴集团）；Salesforce Research（赛富时研究院）；NC State University（北卡罗来纳州立大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Winsense Technology Co. Ltd（北京文因互联科技有限公司）
Chinese traditional painting is one of the most historical artworks in the world. It is very popular in Eastern and Southeast Asia due to being aesthetically appealing. Compared with western artistic painting, it is usually more visually abstract and textureless. Recently, neural network based style transfer methods have shown promising and appealing results which are mainly focused on western painting. It remains a challenging problem to preserve abstraction in neural style transfer. In this paper, we present a Neural Abstract Style Transfer method for Chinese traditional painting. It learns …
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of AI, UCAS（中国科学院大学人工智能学院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Kuaishou Technology（快手科技）
The artistic style within a painting is the means of expression, which includes not only the painting material, colors, and brushstrokes, but also the high-level attributes including semantic elements, object shapes, etc. Previous arbitrary example-guided artistic image generation methods often fail to control shape changes or convey elements. The pre-trained text-to-image synthesis diffusion probabilistic models have achieved remarkable quality, but it often requires extensive textual descriptions to accurately portray attributes of a particular painting. We believe that the uniqueness of an …
FoolSDEdit: Deceptively Steering Your Edits Towards Targeted Attribute-aware Distribution
Zhejiang University（浙江大学）；Nanyang Technological University（南洋理工大学）；Institute of High Performance Computing (IHPC)（高性能计算研究所）；Centre for Frontier AI Research (CFAR)（前沿人工智能研究中心）
Beyond Binary Preference: Aligning Diffusion Models to Fine-grained Criteria by Decoupling Attributes
Zhejiang University（浙江大学）；University of Electronic Science and Technology of China（电子科技大学）；Peking University（北京大学）；University of Nottingham Ningbo China（诺丁汉大学宁波分校）
Digital art synthesis is receiving increasing attention in the multimedia community because of engaging the public with art effectively. Current digital art synthesis methods usually use single-modality inputs as guidance, thereby limiting the expressiveness of the model and the diversity of generated results. To solve this problem, we propose the multimodal guided artwork diffusion (MGAD) model, which is a diffusion-based digital artwork generation approach that utilizes multimodal prompts as guidance to control the classifier-free diffusion model. Additionally, the contrastive language-image…
Although significant progress has been made in synthesizing high-quality and visually realistic face images by unconditional Generative Adversarial Networks (GANs), there still lacks of control over the generation process in order to achieve semantic face editing. In addition, it remains very challenging to maintain other face information untouched while editing the target attributes. In this paper, we propose a novel learning framework, called GuidedStyle, to achieve semantic face editing on StyleGAN by guiding the image generation process with a knowledge network. Furthermore, we allow an at…
In this work we focused on GAN-based solution for the attribute guided face synthesis. Previous works exploited GANs for generation of photo-realistic face images and did not pay attention to the question of diversity of the resulting images. The proposed solution in its turn introducing novel latent space of unit complex numbers is able to provide the diversity on the "birthday paradox" score 3 times higher than the size of the training dataset. It is important to emphasize that our result is shown on relatively small dataset (20k samples vs 200k) while preserving photo-realistic properties o…
Indian Statistical Institute（印度统计研究所）；Østfold University College（东福尔德大学学院）
The performance of facial super-resolution methods relies on their ability to recover facial structures and salient features effectively. Even though the convolutional neural network and generative adversarial network-based methods deliver impressive performances on face hallucination tasks, the ability to use attributes associated with the low-resolution images to improve performance is unsatisfactory. In this paper, we propose an Attribute Guided Attention Generative Adversarial Network which employs novel attribute guided attention (AGA) modules to identify and focus the generation process …
University of British Columbia（不列颠哥伦比亚大学）；Bank of Montreal（蒙特利尔银行）
Recent approaches have achieved great success in image generation from structured inputs, e.g., semantic segmentation, scene graph or layout. Although these methods allow specification of objects and their locations at image-level, they lack the fidelity and semantic control to specify visual appearance of these objects at an instance-level. To address this limitation, we propose a new image generation method that enables instance-level attribute control. Specifically, the input to our attribute-guided generative model is a tuple that contains: (1) object bounding boxes, (2) object categories …
Computer Vision Lab ETH Zürich（苏黎世联邦理工学院计算机视觉实验室）；KU Leuven（鲁汶大学）
Attribute image manipulation has been a very active topic since the introduction of Generative Adversarial Networks (GANs). Exploring the disentangled attribute space within a transformation is a very challenging task due to the multiple and mutually-inclusive nature of the facial images, where different labels (eyeglasses, hats, hair, identity, etc.) can co-exist at the same time. Several works address this issue either by exploiting the modality of each domain/attribute using a conditional random vector noise, or extracting the modality from an exemplary image. However, existing methods cann…
Image inpainting is a technique of completing missing pixels such as occluded region restoration, distracting objects removal, and facial completion. Among these inpainting tasks, facial completion algorithm performs face inpainting according to the user direction. Existing approaches require delicate and well controlled input by the user, thus it is difficult for an average user to provide the guidance sufficiently accurate for the algorithm to generate desired results. To overcome this limitation, we propose an alternative user-guided inpainting architecture that manipulates facial attribute…
Face Animation with an Attribute-Guided Diffusion Model
Beihang University（北京航空航天大学）；Shenzhen Institutes of Advanced Technology, University of Chinese Academy of Sciences（中国科学院大学深圳先进技术研究院）；Zhongguancun Laboratory（中关村实验室）
Paintings and Drawings Aesthetics Assessment with Rich Attributes for Various Artistic Categories
Beijing Electronic Science and Technology Institute（北京电子科技学院）；Central Academy of Fine Arts（中央美术学院）；Beijing University of Technology（北京工业大学）；University of Science and Technology of China（中国科学技术大学）；Beijing Institute of Fashion Technology（北京服装学院）
Oxia Palus（奥克夏帕鲁斯公司）；University College London（伦敦大学学院）；Cambridge University（剑桥大学）
The automatic analysis of fine art paintings presents a number of novel technical challenges to artificial intelligence, computer vision, machine learning, and knowledge representation quite distinct from those arising in the analysis of traditional photographs. The most important difference is that many realist paintings depict stories or episodes in order to convey a lesson, moral, or meaning. One early step in automatic interpretation and extraction of meaning in artworks is the identifications of figures (actors). In Christian art, specifically, one must identify the actors in order to ide…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；DiDi Labs（滴滴实验室）
We propose a novel approach, MUSE, to illustrate textual attributes visually via portrait generation. MUSE takes a set of attributes written in text, in addition to facial features extracted from a photo of the subject as input. We propose 11 attribute types to represent inspirations from a subject's profile, emotion, story, and environment. We propose a novel stacked neural network architecture by extending an image-to-image generative model to accept textual attributes. Experiments show that our approach significantly outperforms several state-of-the-art methods without using textual attribu…
Xiamen University（厦门大学）；Xi’an Jiaotong University（西安交通大学）；University of Chinese Academy of Sciences（中国科学院大学）
Unpaired Image-to-Image Translation (UIT) focuses on translating images among different domains by using unpaired data, which has received increasing research focus due to its practical usage. However, existing UIT schemes defect in the need of supervised training, as well as the lack of encoding domain information. In this paper, we propose an Attribute Guided UIT model termed AGUIT to tackle these two challenges. AGUIT considers multi-modal and multi-domain tasks of UIT jointly with a novel semi-supervised setting, which also merits in representation disentanglement and fine control of outpu…
LatRef-Diff: Latent and Reference-Guided Diffusion for Facial Attribute Editing and Style Manipulation
GuangDong Province Key Lab of Information Security Technology and School of Computer Science and Engineering（广东信息安全技术重点实验室和计算机科学与工程学院）；School of Cyber Science and Technology, Shenzhen Campus（深圳校区网络科学与技术学院）；Guangdong Laboratory of Machine Perception and Intelligent Computing, Faculty of Engineering（广东机器感知与智能计算实验室，工程学院）
Overcoming False Illusions in Real-World Face Restoration with Multi-Modal Guided Diffusion Model
Xidian University（西安电子科技大学）；The University of Sydney（悉尼大学）；Shanghai Jiao Tong University（上海交通大学）；State Key Laboratory of Integrated Services Networks（集成服务网络国家重点实验室）
University at Albany, State University of New York, USA（纽约州立大学奥尔巴尼分校）；Microsoft Corporation, Redmond, USA（微软公司雷德蒙德分部）；Samsung Research America AI Center, USA（三星美国研究公司人工智能中心）；University at Buffalo, State University of New York, USA（纽约州立大学布法罗分校）
We present Mask-guided Generative Adversarial Network (MagGAN) for high-resolution face attribute editing, in which semantic facial masks from a pre-trained face parser are used to guide the fine-grained image editing process. With the introduction of a mask-guided reconstruction loss, MagGAN learns to only edit the facial parts that are relevant to the desired attribute changes, while preserving the attribute-irrelevant regions (e.g., hat, scarf for modification `To Bald'). Further, a novel mask-guided conditioning strategy is introduced to incorporate the influence region of each attribute c…
LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model
KTH Royal Institute of Technology（瑞典皇家理工学院）；Jilin University（吉林大学）；Tokyo Institute of Technology（东京工业大学）；Japan Advanced Institute of Science and Technology (JAIST)（日本先进科学技术学院）
Image-to-image reconstruction problems with free or inexpensive metadata in the form of class labels appear often in biological and medical image domains. Existing text-guided or style-transfer image-to-image approaches do not translate to datasets where additional information is provided as discrete classes. We introduce and implement a model which combines image-to-image and class-guided denoising diffusion probabilistic models. We train our model on a real-world dataset of microscopy images used for drug discovery, with and without incorporating metadata labels. By exploring the properties …
APDDv2: Aesthetics of Paintings and Drawings Dataset with Artist Labeled Scores and Comments
Beijing Electronic Science and Technology Institute（北京电子科技学院）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；Central Academy of Fine Arts（中央美术学院）；University of Science and Technology of China（中国科学技术大学）；Beijing University of Technology（北京工业大学）
University of Trento（特伦托大学）；Huazhong University of Science and Technology（华中科技大学）；EPFL（洛桑联邦理工学院）；University of Oxford（牛津大学）；University of Michigan（密歇根大学）；Texas State University（得克萨斯州立大学）
Facial attributes are important since they provide a detailed description and determine the visual appearance of human faces. In this paper, we aim at converting a face image to a sketch while simultaneously generating facial attributes. To this end, we propose a novel Attribute-Guided Sketch Generative Adversarial Network (ASGAN) which is an end-to-end framework and contains two pairs of generators and discriminators, one of which is used to generate faces with attributes while the other one is employed for image-to-sketch translation. The two generators form a W-shaped network (W-net) and th…
Imperial College London（伦敦帝国学院）；Queen Mary University of London（伦敦玛丽女王大学）；EPFL（洛桑联邦理工学院）；University of Athens（雅典大学）
Generative Adversarial Networks (GANs) are the driving force behind the state-of-the-art in image generation. Despite their ability to synthesize high-resolution photo-realistic images, generating content with on-demand conditioning of different granularity remains a challenge. This challenge is usually tackled by annotating massive datasets with the attributes of interest, a laborious task that is not always a viable option. Therefore, it is vital to introduce control into the generation process of unsupervised generative models. In this work, we focus on controllable image generation by leve…
This work presents a generative modeling approach based on successive subspace learning (SSL). Unlike most generative models in the literature, our method does not utilize neural networks to analyze the underlying source distribution and synthesize images. The resulting method, called the progressive attribute-guided extendable robust image generative (PAGER) model, has advantages in mathematical transparency, progressive content generation, lower training time, robust performance with fewer training samples, and extendibility to conditional image generation. PAGER consists of three modules: c…
The Australian National University（澳大利亚国立大学）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）
Controllable image synthesis with user scribbles has gained huge public interest with the recent advent of text-conditioned latent diffusion models. The user scribbles control the color composition while the text prompt provides control over the overall image semantics. However, we note that prior works in this direction suffer from an intrinsic domain shift problem, wherein the generated outputs often lack details and resemble simplistic representations of the target domain. In this paper, we propose a novel guided image synthesis framework, which addresses this problem by modeling the output…
Automatic art analysis has been mostly focused on classifying artworks into different artistic styles. However, understanding an artistic representation involves more complex processes, such as identifying the elements in the scene or recognizing author influences. We present SemArt, a multi-modal dataset for semantic art understanding. SemArt is a collection of fine-art painting images in which each image is associated to a number of attributes and a textual artistic comment, such as those that appear in art catalogues or museum collections. To evaluate semantic art understanding, we envisage…
School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；NLPR, Institute of Automation, CAS（中国科学院自动化研究所模式识别国家重点实验室）；CASIA-LLVision Joint Lab（中科院自动化所-亮亮视野联合实验室）；Youtu Lab, Tencent（腾讯优图实验室）
Arbitrary style transfer is a significant topic with research value and application prospect. A desired style transfer, given a content image and referenced style painting, would render the content image with the color tone and vivid stroke patterns of the style painting while synchronously maintaining the detailed content structure information. Style transfer approaches would initially learn content and style representations of the content and style references and then generate the stylized images guided by these representations. In this paper, we propose the multi-adaptation network which in…
University of Trento（特伦托大学）；FBK（布鲁诺·凯斯勒基金会）；The Chinese University of Hong Kong（香港中文大学）；Tencent AI Lab（腾讯人工智能实验室）；Inria Grenoble Rhône-Alpes（法国国家信息与自动化研究所格勒诺布尔罗讷-阿尔卑斯中心）；Huawei Research（华为研究院）
Manipulating visual attributes of images through human-written text is a very challenging task. On the one hand, models have to learn the manipulation without the ground truth of the desired output. On the other hand, models have to deal with the inherent ambiguity of natural language. Previous research usually requires either the user to describe all the characteristics of the desired image or to use richly-annotated image captioning datasets. In this work, we propose a novel unsupervised approach, based on image-to-image translation, that alters the attributes of a given image through a comm…
SEDDI（SEDDI 公司）；Universidad Carlos III de Madrid（马德里卡洛斯三世大学）；Universidad Rey Juan Carlos（胡安·卡洛斯国王大学）
We present a deep learning-based method for propagating spatially-varying visual material attributes (e.g. texture maps or image stylizations) to larger samples of the same or similar materials. For training, we leverage images of the material taken under multiple illuminations and a dedicated data augmentation policy, making the transfer robust to novel illumination conditions and affine deformations. Our model relies on a supervised image-to-image translation framework and is agnostic to the transferred domain; we showcase a semantic segmentation, a normal map, and a stylization. Following a…
Microsoft Research（微软研究院）；Shanghai Jiao Tong University（上海交通大学）
We propose a new technique for visual attribute transfer across images that may have very different appearance but have perceptually similar semantic structure. By visual attribute transfer, we mean transfer of visual information (such as color, tone, texture, and style) from one image to another. For example, one image could be that of a painting or a sketch while the other is a photo of a real scene, and both depict the same type of scene. Our technique finds semantically-meaningful dense correspondences between two input images. To accomplish this, it adapts the notion of "image analogy" wi…
University of Oxford（牛津大学）；University of Hong Kong（香港大学）
The goal of our paper is to semantically edit parts of an image matching a given text that describes desired attributes (e.g., texture, colour, and background), while preserving other contents that are irrelevant to the text. To achieve this, we propose a novel generative adversarial network (ManiGAN), which contains two key components: text-image affine combination module (ACM) and detail correction module (DCM). The ACM selects image regions relevant to the given text and then correlates the regions with corresponding semantic words for effective manipulation. Meanwhile, it encodes original …
Facial sketches drawn by artists are widely used for visual identification applications and mostly by law enforcement agencies, but the quality of these sketches depend on the ability of the artist to clearly replicate all the key facial features that could aid in capturing the true identity of a subject. Recent works have attempted to synthesize these sketches into plausible visual images to improve visual recognition and identification. However, synthesizing photo-realistic images from sketches proves to be an even more challenging task, especially for sensitive applications such as suspect …
