Image generation from a single image using generative adversarial networks is quite interesting due to the realism of generated images. However, recent approaches need improvement for such realistic and diverse image generation, when the global context of the image is important such as in face, animal, and architectural image generation. This is mainly due to the use of fewer convolutional layers for mainly capturing the patch statistics and, thereby, not being able to capture global statistics very well. We solve this problem by using attention blocks at selected scales and feeding a random Gaussian blurred image to the discriminator for training. Our results are visually better than the state-of-the-art particularly in generating images that require global context. The diversity of our image generation, measured using the average standard deviation of pixels, is also better.
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
We show how to learn a map that takes a content code, derived from a face image, and a randomly chosen style code to an anime image. We derive an adversarial loss from our simple and effective definitions of style and content. This adversarial loss guarantees the map is diverse -- a very wide range of anime can be produced from a single content code. Under plausible assumptions, the map is not just diverse, but also correctly represents the probability of an anime, conditioned on an input face. In contrast, current multimodal generation procedures cannot capture the complex styles that appear …
We present GLASS, a method for Global and Local Action-driven Sequence Synthesis. GLASS is a generative model that is trained on video sequences in an unsupervised manner and that can animate an input image at test time. The method learns to segment frames into foreground-background layers and to generate transitions of the foregrounds over time through a global and local action representation. Global actions are explicitly related to 2D shifts, while local actions are instead related to (both geometric and photometric) local deformations. GLASS uses a recurrent neural network to transition be…
DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation
FNii-Shenzhen（深圳未来网络研究院）；SSE, CUHK-Shenzhen（香港中文大学（深圳）理工学院）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；Baoji University of Arts and Sciences（宝鸡文理学院）；Sun Yat-sen University（中山大学）
Generating high-quality and diverse human images is an important yet challenging task in vision and graphics. However, existing generative models often fall short under the high diversity of clothing shapes and textures. Furthermore, the generation process is even desired to be intuitively controllable for layman users. In this work, we present a text-driven controllable framework, Text2Human, for a high-quality and diverse human generation. We synthesize full-body human images starting from a given human pose with two dedicated steps. 1) With some texts describing the shapes of clothes, the g…
Center For Automation Research, University of Maryland, College Park（马里兰大学帕克分校自动化研究中心）；Microsoft Research（微软研究院）
We present FusedGAN, a deep network for conditional image synthesis with controllable sampling of diverse images. Fidelity, diversity and controllable sampling are the main quality measures of a good image generation model. Most existing models are insufficient in all three aspects. The FusedGAN can perform controllable sampling of diverse images with very high fidelity. We argue that controllability can be achieved by disentangling the generation process into various stages. In contrast to stacked GANs, where multiple stages of GANs are trained separately with full supervision of labeled inte…
Georgia Institute of Technology（佐治亚理工学院）；University of Surrey（萨里大学）；Adobe Inc.（奥多比公司）
We present CoGS, a novel method for the style-conditioned, sketch-driven synthesis of images. CoGS enables exploration of diverse appearance possibilities for a given sketched object, enabling decoupled control over the structure and the appearance of the output. Coarse-grained control over object structure and appearance are enabled via an input sketch and an exemplar "style" conditioning image to a transformer-based sketch and style encoder to generate a discrete codebook representation. We map the codebook representation into a metric space, enabling fine-grained control over selection and …
Pain in 3D: Generating Controllable Synthetic Faces for Automated Pain Assessment
Vector Institute（向量研究所）；KITE Research Institute（KITE研究机构）；University Health Network（大学健康网络）；Department of Computer Science, University of Toronto（多伦多大学计算机科学系）；Department of Medical Imaging, University of Toronto（多伦多大学医学成像系）；Institute of Biomedical Engineering, University of Toronto（多伦多大学生物医学工程研究所）
Institute of Computer Science and Technology, Peking University（北京大学计算机科学技术研究所）；Texas A&M University（德克萨斯农工大学）；Adobe Research（奥多比研究中心）
Artistic text style transfer is the task of migrating the style from a source image to the target text to create artistic typography. Recent style transfer methods have considered texture control to enhance usability. However, controlling the stylistic degree in terms of shape deformation remains an important open challenge. In this paper, we present the first text style transfer network that allows for real-time control of the crucial stylistic degree of the glyph through an adjustable parameter. Our key contribution is a novel bidirectional shape matching framework to establish an effective …
DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenes
Huazhong University of Science and Technology（华中科技大学）；National Key Laboratory of Multispectral Information Intelligent Processing Technology（多光谱信息智能处理技术国家重点实验室）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）
CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis
Departments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center（融合医学部门、首尔大学医学院、Asan医院）；Department of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center（耳鼻喉科-头颈外科部门、首尔大学医学院、Asan医院）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Controllable semantic image editing enables a user to change entire image attributes with a few clicks, e.g., gradually making a summer scene look like it was taken in winter. Classic approaches for this task use a Generative Adversarial Net (GAN) to learn a latent space and suitable latent-space transformations. However, current approaches often suffer from attribute edits that are entangled, global image identity changes, and diminished photo-realism. To address these concerns, we learn multiple attribute transformations simultaneously, integrate attribute regression into the training of tra…
We present a neural rendering framework that maps a voxelized scene into a high quality image. Highly-textured objects and scene element interactions are realistically rendered by our method, despite having a rough representation as an input. Moreover, our approach allows controllable rendering: geometric and appearance modifications in the input are accurately propagated to the output. The user can move, rotate and scale an object, change its appearance and texture or modify the position of the light and all these edits are represented in the final rendering. We demonstrate the effectiveness …
University of California, Merced（加利福尼亚大学默塞德分校）；National Tsing Hua University（台湾清华大学）；Google Research（谷歌研究院）
Flexible user controls are desirable for content creation and image editing. A semantic map is commonly used intermediate representation for conditional image generation. Compared to the operation on raw RGB pixels, the semantic map enables simpler user modification. In this work, we specifically target at generating semantic maps given a label-set consisting of desired categories. The proposed framework, SegVAE, synthesizes semantic maps in an iterative manner using conditional variational autoencoder. Quantitative and qualitative experiments demonstrate that the proposed model can generate r…
Erase but Preserve: Controllable Removal of Copyrighted Animation Characters via Optimized Semantic Anchors
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
SceneX: Procedural Controllable Large-scale Scene Generation
University of Chinese Academy of Sciences（中国科学院大学）；Centre for Artificial Intelligence and Robotics（人工智能与机器人研究中心）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Shandong University（山东大学）；University of Science and Technology Beijing（北京科技大学）；China University of Geosciences Beijing（中国地质大学（北京））
Controllable Diverse Sampling for Diffusion Based Motion Behavior Forecasting
Leibniz University Hannover（汉诺威莱布尼茨大学）；Institute of Cartography and Geoinformatics（制图与地理信息研究所）；University of Twente（特温特大学）；Faculty of Geo-Information Science and Earth Observation（地理信息科学与地球观测学院）
Despite the remarkable progress of image captioning, existing captioners typically lack the controllable capability to generate desired image captions, e.g., describing the image in a rough or detailed manner, in a factual or emotional view, etc. In this paper, we show that a unified model is qualified to perform well in diverse domains and freely switch among multiple styles. Such a controllable capability is achieved by embedding the prompt learning into the image captioning framework. To be specific, we design a set of prompts to fine-tune the pre-trained image captioner. These prompts allo…
ArtFormer: Controllable Generation of Diverse 3D Articulated Objects
Xiamen University Malaysia（马来西亚厦门大学）；Renmin University of China（中国人民大学）；Tsinghua University（清华大学）；Southern University of Science and Technology（南方科技大学）；University of Minnesota-Twin Cities（明尼苏达大学双城分校）；Henan RunTai Digital Technology Group Co., Ltd.（河南润泰数字科技集团有限公司）
Google（谷歌）；Carnegie Mellon University（卡内基梅隆大学）；Georgia Institute of Technology（佐治亚理工学院）
Creating visual layouts is a critical step in graphic design. Automatic generation of such layouts is essential for scalable and diverse visual designs. To advance conditional layout generation, we introduce BLT, a bidirectional layout transformer. BLT differs from previous work on transformers in adopting non-autoregressive transformers. In training, BLT learns to predict the masked attributes by attending to surrounding attributes in two directions. During inference, BLT first generates a draft layout from the input and then iteratively refines it into a high-quality layout by masking out lo…
Shandong University（山东大学）；Tencent AI Lab（腾讯人工智能实验室）
In this paper, we propose to guide the video caption generation with Part-of-Speech (POS) information, based on a gated fusion of multiple representations of input videos. We construct a novel gated fusion network, with one particularly designed cross-gating (CG) block, to effectively encode and fuse different types of representations, e.g., the motion and content features of an input video. One POS sequence generator relies on this fused representation to predict the global syntactic structure, which is thereafter leveraged to guide the video captioning generation and control the syntax of th…
Shanghai Jiao Tong University（上海交通大学）；The Chinese University of Hong Kong（香港中文大学）；Heriot-Watt University（赫瑞瓦特大学）；Tencent（腾讯）
This paper focuses on self-supervised video representation learning. Most existing approaches follow the contrastive learning pipeline to construct positive and negative pairs by sampling different clips. However, this formulation tends to bias to static background and have difficulty establishing global temporal structures. The major reason is that the positive pairs, i.e., different clips sampled from the same video, have limited temporal receptive field, and usually share similar background but differ in motions. To address these problems, we propose a framework to jointly utilize local cli…
Recent breakthroughs in text-guided image generation have led to remarkable progress in the field of 3D synthesis from text. By optimizing neural radiance fields (NeRF) directly from text, recent methods are able to produce remarkable results. Yet, these methods are limited in their control of each object's placement or appearance, as they represent the scene as a whole. This can be a major issue in scenarios that require refining or manipulating objects in the scene. To remedy this deficit, we propose a novel GlobalLocal training framework for synthesizing a 3D scene using object proxies. A p…
State-of-the-art image captioners can generate accurate sentences to describe images in a sequence to sequence manner without considering the controllability and interpretability. This, however, is far from making image captioning widely used as an image can be interpreted in infinite ways depending on the target and the context at hand. Achieving controllability is important especially when the image captioner is used by different people with different way of interpreting the images. In this paper, we introduce a novel framework for image captioning which can generate diverse descriptions by …
