We introduce an Extended Textual Conditioning space in text-to-image models, referred to as $P+$. This space consists of multiple textual conditions, derived from per-layer prompts, each corresponding to a layer of the denoising U-net of the diffusion model.
We show that the extended space provides greater disentangling and control over image synthesis. We further introduce Extended Textual Inversion (XTI), where the images are inverted into $P+$, and represented by per-layer tokens.
We show that XTI is more expressive and precise, and converges faster than the original Textual Inversion (TI) space. The extended inversion method does not involve any noticeable trade-off between reconstruction and editability and induces more regular inversions.
We conduct a series of extensive experiments to analyze and understand the properties of the new space, and to showcase the effectiveness of our method for personalizing text-to-image models. Furthermore, we utilize the unique properties of this space to achieve previously unattainable results in object-style mixing using text-to-image models. Project page: https://prompt-plus.github.io
Conditioning Matters: Stabilizing Inversion and Attention in Diffusion Image Editing
State Key Laboratory of Blockchain and Data Security, Zhejiang University（浙江大学区块链与数据安全全国重点实验室）；HangZhou High-Tech Zong (Binjiang) Institute of Blockchain and Data Security（杭州高新技术产业开发区（滨江）区块链与数据安全研究院）；College of Computer Science, Zhejiang University（浙江大学计算机科学与技术学院）；University at Buffalo, State University of New York（纽约州立大学布法罗分校）
Generative Adversarial Networks (GANs) have proven successful for unsupervised image generation. Several works extended GANs to image inpainting by conditioning the generation with parts of the image one wants to reconstruct. However, these methods have limitations in settings where only a small subset of the image pixels is known beforehand. In this paper, we study the effectiveness of conditioning GANs by adding an explicit regularization term to enforce pixel-wise conditions when very few pixel values are provided. In addition, we also investigate the influence of this regularization term o…
ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models
Institute of Automation, CAS（中国科学院自动化研究所）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）；Kuaishou Technology（快手科技）；National Cheng-Kung University（国立成功大学）；University of Konstanz（康斯坦茨大学）
Verily Life Sciences（Verily生命科学）；Technion（以色列理工学院）；MIT（麻省理工学院）
Training a generative model on a single image has drawn significant attention in recent years. Single image generative methods are designed to learn the internal patch distribution of a single natural image at multiple scales. These models can be used for drawing diverse samples that semantically resemble the training image, as well as for solving many image editing and restoration tasks that involve that particular image. Here, we introduce an extended framework, which allows to simultaneously learn the internal distributions of several images, by using a single model with spatially varying i…
EditTransfer++: Toward Faithful and Efficient Visual-Prompt-Guided Image Editing
School of Information and Communication Engineering and the State Key Laboratory of Media Convergence and Communication, Communication University of China（信息与通信工程学院和媒体融合与通信国家重点实验室，中国传媒大学）；ShowLab, National University of Singapore（新加坡国立大学ShowLab）
Most methods for conditional video synthesis use a single modality as the condition. This comes with major limitations. For example, it is problematic for a model conditioned on an image to generate a specific motion trajectory desired by the user since there is no means to provide motion information. Conversely, language information can describe the desired motion, while not precisely defining the content of the video. This work presents a multimodal video generation framework that benefits from text and images provided jointly or separately. We leverage the recent progress in quantized repre…
Exploring Motion-Language Alignment for Text-driven Motion Generation
Department of Automation, University of Science and Technology of China（中国科学技术大学自动化系）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，北京通用人工智能研究院）
University of Amsterdam（阿姆斯特丹大学）；Dutch National Police（荷兰国家警察）
In recent years, Generative Adversarial Networks (GANs) have improved steadily towards generating increasingly impressive real-world images. It is useful to steer the image generation process for purposes such as content creation. This can be done by conditioning the model on additional information. However, when conditioning on additional information, there still exists a large set of images that agree with a particular conditioning. This makes it unlikely that the generated image is exactly as envisioned by a user, which is problematic for practical content creation scenarios such as generat…
Generative Adversarial Networks (GANs) have proven successful for unsupervised image generation. Several works have extended GANs to image inpainting by conditioning the generation with parts of the image to be reconstructed. Despite their success, these methods have limitations in settings where only a small subset of the image pixels is known beforehand. In this paper we investigate the effectiveness of conditioning GANs when very few pixel values are provided. We propose a modelling framework which results in adding an explicit cost term to the GAN objective function to enforce pixel-wise c…
OmniControlNet: Dual-stage Integration for Conditional Image Generation
Tsinghua University（清华大学）；University of Science and Technology of China（中国科学技术大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；Princeton University（普林斯顿大学）
Computer Vision Center, Universitat Autònoma de Barcelona（巴塞罗那自治大学计算机视觉中心）；omni:us（omni:us 公司）
Although current image generation methods have reached impressive quality levels, they are still unable to produce plausible yet diverse images of handwritten words. On the contrary, when writing by hand, a great variability is observed across different writers, and even when analyzing words scribbled by the same individual, involuntary variations are conspicuous. In this work, we take a step closer to producing realistic and varied artificially rendered handwritten words. We propose a novel method that is able to produce credible handwritten word images by conditioning the generative process …
Deep learning techniques are increasingly being considered for geological applications where -- much like in computer vision -- the challenges are characterized by high-dimensional spatial data dominated by multipoint statistics. In particular, a novel technique called generative adversarial networks has been recently studied for geological parametrization and synthesis, obtaining very impressive results that are at least qualitatively competitive with previous methods. The method obtains a neural network parametrization of the geology -- so-called a generator -- that is capable of reproducing…
FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidance
AI Research（360人工智能研究院）；Beijing University of Technology（北京理工大学）；Wuhan University（武汉大学）；Sun Yat-sen University（中山大学）；University of Science and Technology of China（中国科学技术大学）；Zhejiang University（浙江大学）
Ahsanullah University of Science and Technology（阿赫桑拉工程与技术大学）
Faces and their expressions are one of the potent subjects for digital images. Detecting emotions from images is an ancient task in the field of computer vision; however, performing its reverse -- synthesizing facial expressions from images -- is quite new. Such operations of regenerating images with different facial expressions, or altering an existing expression in an image require the Generative Adversarial Network (GAN). In this paper, we aim to change the facial expression in an image using GAN, where the input image with an initial expression (i.e., happy) is altered to a different expre…
We introduce X&Fuse, a general approach for conditioning on visual information when generating images from text. We demonstrate the potential of X&Fuse in three different text-to-image generation scenarios. (i) When a bank of images is available, we retrieve and condition on a related image (Retrieve&Fuse), resulting in significant improvements on the MS-COCO benchmark, gaining a state-of-the-art FID score of 6.65 in zero-shot settings. (ii) When cropped-object images are at hand, we utilize them and perform subject-driven generation (Crop&Fuse), outperforming the textual inversion method whil…
MedSyn: Text-guided Anatomy-aware Synthesis of High-Fidelity 3D CT Images
Boston University（波士顿大学）；Stanford University（斯坦福大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Pittsburgh Medical Center（匹兹堡大学医学中心）；University of Pittsburgh（匹兹堡大学）
Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing
Media Integration and Communication Center (MICC), University of Florence（佛罗伦萨大学媒体集成与传播中心（MICC））；University of Florence（佛罗伦萨大学）；University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）；University of Pisa（比萨大学）
University of Technology Sydney（悉尼科技大学）；Indian Statistical Institute（印度统计研究所）；Indian Institute of Technology, Kharagpur（印度理工学院克勒格布尔分校）
Image colorization is a well-known problem in computer vision. However, due to the ill-posed nature of the task, image colorization is inherently challenging. Though several attempts have been made by researchers to make the colorization pipeline automatic, these processes often produce unrealistic results due to a lack of conditioning. In this work, we attempt to integrate textual descriptions as an auxiliary condition, along with the grayscale image that is to be colorized, to improve the fidelity of the colorization process. To the best of our knowledge, this is one of the first attempts to…
