Most existing video diffusion models (VDMs) are limited to mere text conditions. Thereby, they are usually lacking in control over visual appearance and geometry structure of the generated videos. This work presents Moonshot, a new video generation model that conditions simultaneously on multimodal inputs of image and text. The model builts upon a core module, called multimodal video block (MVB), which consists of conventional spatialtemporal layers for representing video features, and a decoupled cross-attention layer to address image and text inputs for appearance conditioning. In addition, we carefully design the model architecture such that it can optionally integrate with pre-trained image ControlNet modules for geometry visual conditions, without needing of extra training overhead as opposed to prior methods. Experiments show that with versatile multimodal conditioning mechanisms, Moonshot demonstrates significant improvement on visual quality and temporal consistency compared to existing models. In addition, the model can be easily repurposed for a variety of generative applications, such as personalized video generation, image animation and video editing, unveiling its potential to serve as a fundamental architecture for controllable video generation. Models will be made public on https://github.com/salesforce/LAVIS.
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Controllable semantic image editing enables a user to change entire image attributes with a few clicks, e.g., gradually making a summer scene look like it was taken in winter. Classic approaches for this task use a Generative Adversarial Net (GAN) to learn a latent space and suitable latent-space transformations. However, current approaches often suffer from attribute edits that are entangled, global image identity changes, and diminished photo-realism. To address these concerns, we learn multiple attribute transformations simultaneously, integrate attribute regression into the training of tra…
University of Maryland, College Park（马里兰大学帕克分校）；PARC, Palo Alto（帕洛阿尔托帕罗奥多研究中心）
Facial expression editing is a challenging task as it needs a high-level semantic understanding of the input face image. In conventional methods, either paired training data is required or the synthetic face resolution is low. Moreover, only the categories of facial expression can be changed. To address these limitations, we propose an Expression Generative Adversarial Network (ExprGAN) for photo-realistic facial expression editing with controllable expression intensity. An expression controller module is specially designed to learn an expressive and compact expression code in addition to the …
Shanghai AI Laboratory（上海人工智能实验室）；Nanyang Technological University（南洋理工大学）；Monash University（莫纳什大学）；SenseTime Research（商汤科技研究院）；Hong Kong University of Science and Technology（香港科技大学）
Recent advances like StyleGAN have promoted the growth of controllable facial editing. To address its core challenge of attribute decoupling in a single latent space, attempts have been made to adopt dual-space GAN for better disentanglement of style and content representations. Nonetheless, these methods are still incompetent to obtain plausible editing results with high controllability, especially for complicated attributes. In this study, we highlight the importance of interaction in a dual-space GAN for more controllable editing. We propose TransEditor, a novel Transformer-based framework …
Recent studies have shown that StyleGANs provide promising prior models for downstream tasks on image synthesis and editing. However, since the latent codes of StyleGANs are designed to control global styles, it is hard to achieve a fine-grained control over synthesized images. We present SemanticStyleGAN, where a generator is trained to model local semantic parts separately and synthesizes images in a compositional way. The structure and texture of different local parts are controlled by corresponding latent codes. Experimental results demonstrate that our model provides a strong disentanglem…
Controllable-Continuous Color Editing in Diffusion Model via Color Mapping
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（人工智能学院，北京邮电大学）；SketchX, CVSSP, University of Surrey（SketchX、CVSSP、 Surrey大学）
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Texas A&M University（德克萨斯农工大学）
Sketch-based image editing aims to synthesize and modify photos based on the structural information provided by the human-drawn sketches. Since sketches are difficult to collect, previous methods mainly use edge maps instead of sketches to train models (referred to as edge-based models). However, sketches display great structural discrepancy with edge maps, thus failing edge-based models. Moreover, sketches often demonstrate huge variety among different users, demanding even higher generalizability and robustness for the editing model to work. In this paper, we propose Deep Plastic Surgery, a …
Face attribute editing aims to generate faces with one or multiple desired face attributes manipulated while other details are preserved. Unlike prior works such as GAN inversion, which has an expensive reverse mapping process, we propose a simple feed-forward network to generate high-fidelity manipulated faces. By simply employing some existing and easy-obtainable prior information, our method can control, transfer, and edit diverse attributes of faces in the wild. The proposed method can consequently be applied to various applications such as face swapping, face relighting, and makeup transf…
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
The success of Deep Generative Models at high-resolution image generation has led to their extensive utilization for style editing of real images. Most existing methods work on the principle of inverting real images onto their latent space, followed by determining controllable directions. Both inversion of real images and determination of controllable latent directions are computationally expensive operations. Moreover, the determination of controllable latent directions requires additional human supervision. This work aims to explore the efficacy of mask-guided feature modulation in the laten…
CMD: Controllable Multiview Diffusion for 3D Editing and Progressive Generation
The Hong Kong University of Science and Technology（香港科学与技术大学）；Johns Hopkins University（约翰霍普金斯大学）；University of British Columbia（不列颠哥伦比亚大学）；Texas A\&M University（德克萨斯A&M大学）
Adaptive and flexible image editing is a desirable function of modern generative models. In this work, we present a generative model with auto-encoder architecture for per-region style manipulation. We apply a code consistency loss to enforce an explicit disentanglement between content and style latent representations, making the content and style of generated samples consistent with their corresponding content and style references. The model is also constrained by a content alignment loss to ensure the foreground editing will not interfere background contents. As a result, given interested re…
Towards Consistent and Controllable Image Synthesis for Face Editing
Center for Machine Vision and Signal Analysis, Faculty of Information Technology and Electrical Engineering, University of Oulu（机器视觉与信号分析中心，信息科技与电气工程学院，奥卢大学）；Key Laboratory of Child Development and Learning Science of Ministry of Education, School of Biological Sciences and Medical Engineering, Southeast University（教育部儿童发展与学习科学重点实验室，生物科学与医学工程学院，东南大学）
Realistic and Controllable 3D Gaussian-Guided Object Editing for Driving Video Generation
School of Vehicle and Mobility, and State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University（车辆与移动系统学院，智能绿色车辆与移动系统国家重点实验室，清华大学）；WUWEN AI；PhiGent Robotics
DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenes
Huazhong University of Science and Technology（华中科技大学）；National Key Laboratory of Multispectral Information Intelligent Processing Technology（多光谱信息智能处理技术国家重点实验室）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）
DICE: Discrete Inversion Enabling Controllable Editing for Multinomial Diffusion and Masked Generative Models
Rutgers University（新泽西罗格斯大学）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；Red Hat AI Innovation（红帽AI创新）；Google DeepMind（谷歌DeepMind）；NYU（纽约大学）；Walmart Global Tech（沃尔玛全球技术）；NEC Labs America（NEC美国实验室）；Massachusetts Institute of Technology（麻省理工学院）；ANU（澳大利亚国立大学）；UT Arlington（德克萨斯大学阿灵顿分校）
3D-controllable portrait synthesis has significantly advanced, thanks to breakthroughs in generative adversarial networks (GANs). However, it is still challenging to manipulate existing face images with precise 3D control. While concatenating GAN inversion and a 3D-aware, noise-to-image GAN is a straight-forward solution, it is inefficient and may lead to noticeable drop in editing quality. To fill this gap, we propose 3D-FM GAN, a novel conditional GAN framework designed specifically for 3D-controllable face manipulation, and does not require any tuning after the end-to-end learning phase. By…
Visual Prompting for One-shot Controllable Video Editing without Inversion
Singapore University of Technology and Design（新加坡科技设计大学）；Wuhan University（武汉大学）；Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore（新加坡资讯与通信研究院）
Latent space exploration is a technique that discovers interpretable latent directions and manipulates latent codes to edit various attributes in images generated by generative adversarial networks (GANs). However, in previous work, spatial control is limited to simple transformations (e.g., translation and rotation), and it is laborious to identify appropriate latent directions and adjust their parameters. In this paper, we tackle the problem of editing the StyleGAN image layout by annotating the image directly. To do so, we propose an interactive framework for manipulating latent codes in ac…
FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Science（中国科学院大学人工智能学院）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室）；Wuhan AI Research, Wuhan, China（武汉人工智能研究所）
HUG-VAS: A Hierarchical NURBS-Based Generative Model for Aortic Geometry Synthesis and Controllable Editing
Department of Aerospace and Mechanical Engineering, University of Notre Dame（notre dame 大学航空航天与机械工程系）；Sibley School of Mechanical and Aerospace Engineering, Cornell University（cornell 大学机械与航空航天工程学院）；Department of Applied and Computational Mathematics and Statistics, University of Notre Dame（notre dame 大学应用与计算数学与统计学系）
Tampere University（坦佩雷大学）；Aalto University of Technology（阿尔托理工大学）
This paper presents a generic face animator that is able to control the pose and expressions of a given face image. The animation is driven by human interpretable control signals consisting of head pose angles and the Action Unit (AU) values. The control information can be obtained from multiple sources including external driving videos and manual controls. Due to the interpretable nature of the driving signal, one can easily mix the information between multiple sources (e.g. pose from one image and expression from another) and apply selective post-production editing. The proposed face animato…
BlobCtrl: Taming Controllable Blob for Element-level Image Editing
SECE, Peking University（北京大学SECE学院）；The Chinese University of Hong Kong（香港中文大学）；ARC Lab, Tencent（腾讯ARC实验室）；The Hong Kong University of Science and Technology（香港科学与技术大学）；GVC Lab, Great Bay University（Great Bay大学GVC实验室）
ZeroScene: A Zero-Shot Framework for 3D Scene Generation from a Single Image and Controllable Texture Editing
Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））；Pengcheng Laboratory, China（鹏城实验室）；Harbin Institute of Technology, China（哈尔滨工业大学）；Harbin Institute of Technology, Suzhou Research Institute, China（哈尔滨工业大学苏州市研究院）
FaceEditTalker: Controllable Talking Head Generation with Facial Attribute Editing
Xi’an Key Laboratory of Big Data and Intelligent Vision, Xidian University, Xi’an 710071, China（西安大数据与智能视觉重点实验室，西安电子科技大学，西安710071，中国）；Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, Xidian University, Xi’an 710071, China（协同智能系统重点实验室，教育部，西安电子科技大学，西安710071，中国）；School of Computer Science and Technology, Xidian University, Xi’an 710071, China（计算机科学与技术学院，西安电子科技大学，西安710071，中国）
