Conditional generative models typically demand large annotated training sets to achieve high-quality synthesis. As a result, there has been significant interest in designing models that perform plug-and-play generation, i.e., to use a predefined or pretrained model, which is not explicitly trained on the generative task, to guide the generative process (e.g., using language). However, such guidance is typically useful only towards synthesizing high-level semantics rather than editing fine-grained details as in image-to-image translation tasks. To this end, and capitalizing on the powerful fine-grained generative control offered by the recent diffusion-based generative models, we introduce Steered Diffusion, a generalized framework for photorealistic zero-shot conditional image generation using a diffusion model trained for unconditional generation. The key idea is to steer the image generation of the diffusion model at inference time via designing a loss using a pre-trained inverse model that characterizes the conditional task. This loss modulates the sampling trajectory of the diffusion process. Our framework allows for easy incorporation of multiple conditions during inference. We present experiments using steered diffusion on several tasks including inpainting, colorization, text-guided semantic editing, and image super-resolution. Our results demonstrate clear qualitative and quantitative improvements over state-of-the-art diffusion-based plug-and-play models while adding negligible additional computational cost.
Wroclaw University of Science and Technology（弗罗茨瓦夫理工大学）；Tooploox Ltd.（Tooploox有限公司）
Generative models have gained many researchers' attention in the last years resulting in models such as StyleGAN for human face generation or PointFlow for the 3D point cloud generation. However, by default, we cannot control its sampling process, i.e., we cannot generate a sample with a specific set of attributes. The current approach is model retraining with additional inputs and different architecture, which requires time and computational resources. We propose a novel approach that enables to a generation of objects with a given set of attributes without retraining the base model. For this…
Jagiellonian University（雅盖隆大学）；Wroclaw University of Science and Technology（弗罗茨瓦夫科技大学）；Institute of Pharmacology PAS（波兰科学院药理学研究所）；Tooploox
Modern generative models achieve excellent quality in a variety of tasks including image or text generation and chemical molecule modeling. However, existing methods often lack the essential ability to generate examples with requested properties, such as the age of the person in the photo or the weight of the generated molecule. Incorporating such additional conditioning factors would require rebuilding the entire architecture and optimizing the parameters from scratch. Moreover, it is difficult to disentangle selected attributes so that to perform edits of only one attribute while leaving the…
While generative adversarial networks (GANs) have revolutionized machine learning, a number of open questions remain to fully understand them and exploit their power. One of these questions is how to efficiently achieve proper diversity and sampling of the multi-mode data space. To address this, we introduce BasisGAN, a stochastic conditional multi-mode image generator. By exploiting the observation that a convolutional filter can be well approximated as a linear combination of a small set of basis elements, we learn a plug-and-played basis generator to stochastically generate basis elements, …
Geometric Intelligence（几何智能）；Uber AI Labs（优步人工智能实验室）；University of Wyoming（怀俄明大学）；Montreal Institute for Learning Algorithms（蒙特利尔学习算法研究所）；University of Freiburg（弗莱堡大学）
Generating high-resolution, photo-realistic images has been a long-standing goal in machine learning. Recently, Nguyen et al. (2016) showed one interesting way to synthesize novel images by performing gradient ascent in the latent space of a generator network to maximize the activations of one or multiple neurons in a separate classifier network. In this paper we extend this method by introducing an additional prior on the latent code, improving both sample quality and sample diversity, leading to a state-of-the-art generative model that produces high quality images at higher resolutions (227x…
We propose a simple but effective modification of the discriminators, namely measure-conditional discriminators, as a plug-and-play module for different GANs. By taking the generated distributions as part of input so that the target optimum for the discriminator is stationary, the proposed discriminator is more robust than the vanilla one. A variant of the measure-conditional discriminator can also handle multiple target distributions, or act as a surrogate model of statistical distances such as KL divergence with applications to transfer learning.
Morphology-Aware Peptide Discovery via Masked Conditional Generative Modeling
Technical University of Munich（慕尼黑技术大学）；TUM School of Engineering and Design（慕尼黑技术大学工程与设计学院）；Department of Engineering Physics and Computation（工程物理与计算系）；Munich Data Science Institute（慕尼黑数据科学研究所）；AMC（原子模型中心）
Conditional diffusion probabilistic models can model the distribution of natural images and can generate diverse and realistic samples based on given conditions. However, oftentimes their results can be unrealistic with observable color shifts and textures. We believe that this issue results from the divergence between the probabilistic distribution learned by the model and the distribution of natural images. The delicate conditions gradually enlarge the divergence during each sampling timestep. To address this issue, we introduce a new method that brings the predicted samples to the training …
Diffusion-based generative models have achieved remarkable success in image generation. Their guidance formulation allows an external model to plug-and-play control the generation process for various tasks without finetuning the diffusion model. However, the direct use of publicly available off-the-shelf models for guidance fails due to their poor performance on noisy inputs. For that, the existing practice is to fine-tune the guidance models with labeled data corrupted with noises. In this paper, we argue that this practice has limitations in two aspects: (1) performing on inputs with extreme…
Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretization
Harbin Institute Of Technology（哈尔滨工业大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Harbin Institute Of Technology (Shenzhen)（哈尔滨工业大学（深圳））；University of Oxford（牛津大学）
Hong Kong University of Science and Technology（香港科技大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Hong Kong Baptist University（香港浸会大学）
In a nonparametric setting, the causal structure is often identifiable only up to Markov equivalence, and for the purpose of causal inference, it is useful to learn a graphical representation of the Markov equivalence class (MEC). In this paper, we revisit the Greedy Equivalence Search (GES) algorithm, which is widely cited as a score-based algorithm for learning the MEC of the underlying causal structure. We observe that in order to make the GES algorithm consistent in a nonparametric setting, it is not necessary to design a scoring metric that evaluates graphs. Instead, it suffices to plug i…
Plug-and-Play Context Feature Reuse for Efficient Masked Generation
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Computer Science Department, University of California, Los Angeles（加州大学洛杉矶分校计算机科学系）；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学校）
Diffusion Sampling Path Tells More: An Efficient Plug-and-Play Strategy for Sample Filtering
School of Science and Engineering（科学与工程学院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；DAMO Academy, Alibaba Group（阿里达摩院）；School of Artificial Intelligence（人工智能学院）
Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学（KAUST））；Center for Research and Formation in Artificial Intelligence, Universidad de los Andes（安第斯大学人工智能研究与培训中心）；GenAI, Meta（Meta 生成式人工智能部门）
Shortest-Path Flow Matching with Mixture-Conditioned Bases for OOD Generalization to Unseen Conditions
Wellcome Sanger Institute（Wellcome桑格研究所）；Institute for Molecular Medicine Finland (FIMM)（芬兰分子医学研究所）；University of Helsinki（赫尔辛基大学）；School of Computing and Information Systems（计算与信息系统学院）；University of Cambridge（剑桥大学）；Cambridge Centre for AI in Medicine（剑桥人工智能医学中心）；MediaTek Research（MediaTek研究）；Cambridge Stem Cell Institute（剑桥干细胞研究所）
Accelerating Conditional Prompt Learning via Masked Image Modeling for Vision-Language Models
Convergence Research Institute, Sungkyunkwan University（convergence research institute, 首尔大学）；School of Information Technology, Deakin University（信息科技学院, 德金大学）；Department of Electrical and Computer Engineering, Sungkyunkwan University（电气与计算机工程系, 首尔大学）
The Chinese University of Hong Kong (CUHK)（香港中文大学（CUHK））
To synthesize a realistic action sequence based on a single human image, it is crucial to model both motion patterns and diversity in the action video. This paper proposes an Action Conditional Temporal Variational AutoEncoder (ACT-VAE) to improve motion prediction accuracy and capture movement diversity. ACT-VAE predicts pose sequences for an action clips from a single input image. It is implemented as a deep generative model that maintains temporal coherence according to the action category with a novel temporal modeling on latent space. Further, ACT-VAE is a general action sequence predicti…
ReMA: A Training-Free Plug-and-Play Mixing Augmentation for Video Behavior Recognition
University of Science and Technology of China（科学技术大学）；Hefei University of Technology（合肥工业大学）；Hefei Xiaosheng Intelligent Technology Co., Ltd.（合肥小生智能科技有限公司）；Cylingo Group（Cylingo集团）；The University of Electro-Communications（电通大学）
