The goal of this paper is to embed controllable factors, i.e., natural language descriptions, into image-to-image translation with generative adversarial networks, which allows text descriptions to determine the visual attributes of synthetic images. We propose four key components: (1) the implementation of part-of-speech tagging to filter out non-semantic words in the given description, (2) the adoption of an affine combination module to effectively fuse different modality text and image features, (3) a novel refined multi-stage architecture to strengthen the differential ability of discriminators and the rectification ability of generators, and (4) a new structure loss to further improve discriminators to better distinguish real and synthetic images. Extensive experiments on the COCO dataset demonstrate that our method has a superior performance on both visual realism and semantic consistency with given descriptions.
We address the problem of guided image-to-image translation where we translate an input image into another while respecting the constraints provided by an external, user-provided guidance image. Various conditioning methods for leveraging the given guidance image have been explored, including input concatenation , feature concatenation, and conditional affine transformation of feature activations. All these conditioning mechanisms, however, are uni-directional, i.e., no information flow from the input image back to the guidance. To better utilize the constraints of the guidance image, we prese…
Language-Grounded Multi-Domain Image Translation via Semantic Difference Guidance
Department of Artificial Intelligence, Chung-Ang University（人工智能系， Chung-Ang 大学）；Hyundai Mobis；Korea Advanced Institute of Science and Technology（韩国科学技术院）
Auxiliary CycleGAN-guidance for Task-Aware Domain Translation from Duplex to Monoplex IHC Images
AstraZeneca Computational Pathology, Oncology Research and Development（阿斯利康肿瘤研发部计算病理学）；AstraZeneca Translational Pathology, Oncology Research and Development（阿斯利康肿瘤研发部转化病理学）
VinAI Research（VinAI研究院）；FPT University（FPT大学）；University of Arkansas（阿肯色大学）
Image synthesis from corrupted contrasts increases the diversity of diagnostic information available for many neurological diseases. Recently the image-to-image translation has experienced significant levels of interest within medical research, beginning with the successful use of the Generative Adversarial Network (GAN) to the introduction of cyclic constraint extended to multiple domains. However, in current approaches, there is no guarantee that the mapping between the two image domains would be unique or one-to-one. In this paper, we introduce a novel approach to unpaired image-to-image tr…
University of Trento（特伦托大学）；Texas State University（德克萨斯州立大学）；University of Oxford（牛津大学）；Huawei Technologies Ireland（华为技术爱尔兰公司）；Northeastern University（东北大学）；University of Michigan（密歇根大学）
Cross-view image translation is challenging because it involves images with drastically different views and severe deformation. In this paper, we propose a novel approach named Multi-Channel Attention SelectionGAN (SelectionGAN) that makes it possible to generate images of natural scenes in arbitrary viewpoints, based on an image of the scene and a novel semantic map. The proposed SelectionGAN explicitly utilizes the semantic information and consists of two stages. In the first stage, the condition image and the target semantic map are fed into a cycled semantic-guided generation network to pr…
ETH Zurich（苏黎世联邦理工学院）；University of Oxford（牛津大学）；University of Trento（特伦托大学）
We propose a novel model named Multi-Channel Attention Selection Generative Adversarial Network (SelectionGAN) for guided image-to-image translation, where we translate an input image into another while respecting an external semantic guidance. The proposed SelectionGAN explicitly utilizes the semantic guidance information and consists of two stages. In the first stage, the input image and the conditional semantic guidance are fed into a cycled semantic-guided generation network to produce initial coarse results. In the second stage, we refine the initial results by using the proposed multi-sc…
Text-to-Image translation has been an active area of research in the recent past. The ability for a network to learn the meaning of a sentence and generate an accurate image that depicts the sentence shows ability of the model to think more like humans. Popular methods on text to image translation make use of Generative Adversarial Networks (GANs) to generate high quality images based on text input, but the generated images don't always reflect the meaning of the sentence given to the model as input. We address this issue by using a captioning network to caption on generated images and exploit…
Diffusion generative models have recently greatly improved the power of text-conditioned image generation. Existing image generation models mainly include text conditional diffusion model and cross-modal guided diffusion model, which are good at small scene image generation and complex scene image generation respectively. In this work, we propose a simple yet effective approach, namely UPainting, to unify simple and complex scene image generation, as shown in Figure 1. Based on architecture improvements and diverse guidance schedules, UPainting effectively integrates cross-modal guidance from …
Large-scale text-to-image generative models have been a revolutionary breakthrough in the evolution of generative AI, allowing us to synthesize diverse images that convey highly complex visual concepts. However, a pivotal challenge in leveraging such models for real-world content creation tasks is providing users with control over the generated content. In this paper, we present a new framework that takes text-to-image synthesis to the realm of image-to-image translation -- given a guidance image and a target text prompt, our method harnesses the power of a pre-trained text-to-image diffusion …
Diffusion models have recently been shown to generate high-quality synthetic images, especially when paired with a guidance technique to trade off diversity for fidelity. We explore diffusion models for the problem of text-conditional image synthesis and compare two different guidance strategies: CLIP guidance and classifier-free guidance. We find that the latter is preferred by human evaluators for both photorealism and caption similarity, and often produces photorealistic samples. Samples from a 3.5 billion parameter text-conditional diffusion model using classifier-free guidance are favored…
UC Berkeley（加州大学伯克利分校）；Picsart AI Research (PAIR)（Picsart人工智能研究院（PAIR））；University of Oregon（俄勒冈大学）；The University of Hong Kong（香港大学）
Controllable image synthesis models allow creation of diverse images based on text instructions or guidance from a reference image. Recently, denoising diffusion probabilistic models have been shown to generate more realistic imagery than prior methods, and have been successfully demonstrated in unconditional and class-conditional settings. We investigate fine-grained, continuous control of this model class, and introduce a novel unified framework for semantic diffusion guidance, which allows either language or image guidance, or both. Guidance is injected into a pretrained unconditional diffu…
TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；College of Computer Science, Nankai University（南开大学计算机学院）
University of Chicago（芝加哥大学）；Adobe Research（奥多比研究院）；University of Surrey（萨里大学）
We develop an approach for text-to-image generation that embraces additional retrieval images, driven by a combination of implicit visual guidance loss and generative objectives. Unlike most existing text-to-image generation methods which merely take the text as input, our method dynamically feeds cross-modal search results into a unified training stage, hence improving the quality, controllability and diversity of generation results. We propose a novel hypernetwork modulated visual-text encoding scheme to predict the weight update of the encoding layer, enabling effective transfer from visual…
Diffusion-based text-to-image generation models like GLIDE and DALLE-2 have gained wide success recently for their superior performance in turning complex text inputs into images of high quality and wide diversity. In particular, they are proven to be very powerful in creating graphic arts of various formats and styles. Although current models supported specifying style formats like oil painting or pencil drawing, fine-grained style features like color distributions and brush strokes are hard to specify as they are randomly picked from a conditional distribution based on the given text input. …
Diffusion based Zero-shot Medical Image-to-Image Translation for Cross Modality Segmentation
Athinoula A. Martinos Center for Biomedical Imaging（阿西诺拉·A.马蒂诺斯生物医学成像中心）；Massachusetts General Hospital（麻省总医院）；Harvard University（哈佛大学）；Inria centre at Université Côte d’Azur（蔚蓝海岸大学国家信息与自动化研究所中心）；Temple University（天普大学）；Georg-August-University of Göttingen（哥廷根大学）
KU-Leuven/PSI（鲁汶大学/PSI）；TRACE (Toyota Res in Europe)（丰田欧洲研究院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；ETH Zurich（苏黎世联邦理工学院）
Image-to-image translation has recently received significant attention due to advances in deep learning. Most works focus on learning either a one-to-one mapping in an unsupervised way or a many-to-many mapping in a supervised way. However, a more practical setting is many-to-many mapping in an unsupervised way, which is harder due to the lack of supervision and the complex inner- and cross-domain variations. To alleviate these issues, we propose the Exemplar Guided & Semantically Consistent Image-to-image Translation (EGSC-IT) network which conditions the translation process on an exemplar im…
TextGuider: Training-Free Guidance for Text Rendering via Attention Alignment
Interdisciplinary Program in Artificial Intelligence, Seoul National University（人工智能交叉学科项目，首尔国立大学）；Department of Electrical and Computer Engineering, Seoul National University（电子与计算机工程系，首尔国立大学）；Global Technology Research, Samsung Electronics（三星电子全球技术研究）；AIIS, ASRI, INMC, ISRC, Seoul National University（AIIS、ASRI、INMC、ISRC、首尔国立大学）
FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidance
AI Research（360人工智能研究院）；Beijing University of Technology（北京理工大学）；Wuhan University（武汉大学）；Sun Yat-sen University（中山大学）；University of Science and Technology of China（中国科学技术大学）；Zhejiang University（浙江大学）
Dual Orthogonal Guidance for Robust Diffusion-based Handwritten Text Generation
Luleå University of Technology（卢莱大学）；National Technical University of Athens（雅典技术大学）；University of West Attica（西阿提卡大学）；University of Modena and Reggio Emilia（摩德纳和雷吉奥艾米利亚大学）
TeEFusion: Blending Text Embeddings to Distill Classifier-Free Guidance
School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家实验室）；Alibaba International Digital Commerce Group（阿里巴巴国际数字商业集团）
