We present CoGS, a novel method for the style-conditioned, sketch-driven synthesis of images. CoGS enables exploration of diverse appearance possibilities for a given sketched object, enabling decoupled control over the structure and the appearance of the output. Coarse-grained control over object structure and appearance are enabled via an input sketch and an exemplar "style" conditioning image to a transformer-based sketch and style encoder to generate a discrete codebook representation. We map the codebook representation into a metric space, enabling fine-grained control over selection and interpolation between multiple synthesis options before generating the image via a vector quantized GAN (VQGAN) decoder. Our framework thereby unifies search and synthesis tasks, in that a sketch and style pair may be used to run an initial synthesis which may be refined via combination with similar results in a search corpus to produce an image more closely matching the user's intent. We show that our model, trained on the 125 object classes of our newly created Pseudosketches dataset, is capable of producing a diverse gamut of semantic content and appearance styles.
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Texas A&M University（德克萨斯农工大学）
Sketch-based image editing aims to synthesize and modify photos based on the structural information provided by the human-drawn sketches. Since sketches are difficult to collect, previous methods mainly use edge maps instead of sketches to train models (referred to as edge-based models). However, sketches display great structural discrepancy with edge maps, thus failing edge-based models. Moreover, sketches often demonstrate huge variety among different users, demanding even higher generalizability and robustness for the editing model to work. In this paper, we propose Deep Plastic Surgery, a …
Sun Yat-sen University（中山大学）；Nanjing University（南京大学）；Huawei Technologies（华为技术有限公司）
We introduce the first method for automatic image generation from scene-level freehand sketches. Our model allows for controllable image generation by specifying the synthesis goal via freehand sketches. The key contribution is an attribute vector bridged Generative Adversarial Network called EdgeGAN, which supports high visual-quality object-level image content generation without using freehand sketches as training data. We have built a large-scale composite dataset called SketchyCOCO to support and evaluate the solution. We validate our approach on the tasks of both object-level and scene-le…
School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））
Neural style transfer has been demonstrated to be powerful in creating artistic image with help of Convolutional Neural Networks (CNN). However, there is still lack of computational analysis of perceptual components of the artistic style. Different from some early attempts which studied the style by some pre-processing or post-processing techniques, we investigate the characteristics of the style systematically based on feature map produced by CNN. First, we computationally decompose the style into basic elements using not only spectrum based methods including Fast Fourier Transform (FFT), Dis…
Sketch2Human: Deep Human Generation with Disentangled Geometry and Appearance Control
School of Creative Media, City University of Hong Kong（香港城市大学创意媒体学院）；Department of Computer Science & Engineering, HKUST（香港科技大学计算机科学与工程学系）；Shenzhen Research Institute of Big Data, Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）深圳大数据研究院）
Georgia Institute of Technology（佐治亚理工学院）；Adobe Research（Adobe研究院）；Princeton University（普林斯顿大学）
Recently, there have been several promising methods to generate realistic imagery from deep convolutional networks. These methods sidestep the traditional computer graphics rendering pipeline and instead generate imagery at the pixel level by learning from large collections of photos (e.g. faces or bedrooms). However, these methods are of limited utility because it is difficult for a user to control what the network produces. In this paper, we propose a deep adversarial image synthesis architecture that is conditioned on sketched boundaries and sparse color strokes to generate realistic cars, …
SketchVideo: Sketch-based Video Generation and Editing
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Hong Kong University of Science and Technology（香港科技大学）；Kuaishou Technology（快手科技）
Japan Advanced Institute of Science and Technology（日本先进科学技术学院）；The Chinese University of Hong Kong（香港中文大学）；Cornell University（康奈尔大学）；Shanghai Jiao Tong University（上海交通大学）
Center for Research on Intelligent Perception and Computing, NLPR, CASIA（中国科学院自动化研究所模式识别国家重点实验室智能感知与计算研究中心）；University of Chinese Academy of Sciences（中国科学院大学）；Meituandianping Group（美团点评集团）；Artificial Intelligence Research, CAS（中国科学院人工智能研究院）；Center for Excellence in Brain Science and Intelligence Technology, CAS（中国科学院脑科学与智能技术卓越创新中心）
Face portrait editing has achieved great progress in recent years. However, previous methods either 1) operate on pre-defined face attributes, lacking the flexibility of controlling shapes of high-level semantic facial components (e.g., eyes, nose, mouth), or 2) take manually edited mask or sketch as an intermediate representation for observable changes, but such additional input usually requires extra efforts to obtain. To break the limitations (e.g. shape, mask or sketch) of the existing methods, we propose a novel framework termed r-FACE (Reference-guided FAce Component Editing) for diverse…
Recent large-scale generative models learned on big data are capable of synthesizing incredible images yet suffer from limited controllability. This work offers a new generation paradigm that allows flexible control of the output image, such as spatial layout and palette, while maintaining the synthesis quality and model creativity. With compositionality as the core idea, we first decompose an image into representative factors, and then train a diffusion model with all these factors as the conditions to recompose the input. At the inference stage, the rich intermediate representations work as …
We propose a high-quality photo-to-pencil translation method with fine-grained control over the drawing style. This is a challenging task due to multiple stroke types (e.g., outline and shading), structural complexity of pencil shading (e.g., hatching), and the lack of aligned training data pairs. To address these challenges, we develop a two-branch model that learns separate filters for generating sketchy outlines and tonal shading from a collection of pencil drawings. We create training data pairs by extracting clean outlines and tonal illustrations from original pencil drawings using image …
Sketches make an intuitive and powerful visual expression as they are fast executed freehand drawings. We present a method for synthesizing realistic photos from scene sketches. Without the need for sketch and photo pairs, our framework directly learns from readily available large-scale photo datasets in an unsupervised manner. To this end, we introduce a standardization module that provides pseudo sketch-photo pairs during training by converting photos and sketches to a standardized domain, i.e. the edge map. The reduced domain gap between sketch and photo also allows us to disentangle them i…
Nanyang Technological University（南洋理工大学）；Alibaba Group（阿里巴巴集团）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）
We develop a new method for portrait image editing, which supports fine-grained editing of geometries, colors, lights and shadows using a single neural network model. We adopt a novel asymmetric conditional GAN architecture: the generators take the transformed conditional inputs, such as edge maps, color palette, sliders and masks, that can be directly edited by the user; the discriminators take the conditional inputs in the way that can guide controllable image generation more effectively. Taking color editing as an example, we feed color palettes (which can be edited easily) into the generat…
TaleCrafter: Interactive Story Visualization with Multiple Characters
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Tencent AI Lab（腾讯人工智能实验室）；Hong Kong University of Science and Technology（香港科技大学）
NuiWorld: Exploring a Scalable Framework for End-to-End Controllable World Generation
Simon Fraser University（西蒙弗雷泽大学）；National Yang Ming Chiao Tung University（国家阳明交通大学）；Simon Fraser University, CIFAR AI Chair, Amii（西蒙弗雷泽大学、CIFAR人工智能主席、Amii）
Learning from Pattern Completion: Self-supervised Controllable Generation
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Institute of Automation, Chinese Academy of Science（中国科学院自动化研究所）；Peking University（北京大学）；Xi’an Jiaotong University（西安交通大学）；University of Chinese Academy of Science（中国科学院大学）
IMAGGarment: Fine-Grained Garment Generation for Controllable Fashion Design
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；NExT++ Research Centre, National University of Singapore（新加坡国立大学NExT++研究中心）；College of Information Science and Technology and Artificial Intelligence, Nanjing Forestry University（南京林业大学信息科学与技术学院及人工智能学院）
InfiniVerse: Occupancy Guided Unbounded Scene Generation for Autonomous Driving
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanyang Technological University（南洋理工大学）；Huawei Technologies Ltd.（华为技术有限公司）；University of New South Wales（新南威尔士大学）
