Speech Driven Video Editing via an Audio-Conditioned Diffusion Model
University of Galway（高威大学）；University of Wrocław（弗罗茨瓦夫大学）；Wrocław University of Science and Technology（弗罗茨瓦夫理工大学）；Tooploox（Tooploox公司）；Trinity College Dublin（都柏林圣三一学院）
Dan Bigioi, Shubhajit Basak, Michał Stypułkowski, Maciej Zięba, Hugh Jordan, Rachel McDonnell, Peter Corcoran
Taking inspiration from recent developments in visual generative tasks using diffusion models, we propose a method for end-to-end speech-driven video editing using a denoising diffusion model. Given a video of a talking person, and a separate auditory speech recording, the lip and jaw motions are re-synchronized without relying on intermediate structural representations such as facial landmarks or a 3D face model. We show this is possible by conditioning a denoising diffusion model on audio mel spectral features to generate synchronised facial motion. Proof of concept results are demonstrated on both single-speaker and multi-speaker video editing, providing a baseline model on the CREMA-D audiovisual data set. To the best of our knowledge, this is the first work to demonstrate and validate the feasibility of applying end-to-end denoising diffusion models to the task of audio-driven video editing.
Object-AVEdit: An Object-level Audio-Visual Editing Model
Institute of Artificial Intelligence (TeleAI)（人工智能研究院）；China Telecom（中国电信）；Gaoling School of Artificial Intelligence（东城区人工智能学院）；Renmin University of China（中国人民大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Tencent Data Platform（腾讯数据平台）；Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Huawei Noah’s Ark Lab（华为诺亚实验室）；The University of Hong Kong（香港大学）
AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities
Nanjing University（南京大学）；Kuaishou Technology（快手科技）；National University of Singapore（新加坡国立大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation
Zhejiang University（浙江大学）；Tencent Youtu Lab（腾讯优图实验室）；Nanjing University（南京大学）；University of Auckland（奥克兰大学）；Fudan University（复旦大学）；National University of Singapore（新加坡国立大学）
University of Ljubljana（卢布尔雅那大学）；Queensland University of Technology（昆士兰科技大学）
Fashion-image editing represents a challenging computer vision task, where the goal is to incorporate selected apparel into a given input image. Most existing techniques, known as Virtual Try-On methods, deal with this task by first selecting an example image of the desired apparel and then transferring the clothing onto the target person. Conversely, in this paper, we consider editing fashion images with text descriptions. Such an approach has several advantages over example-based virtual try-on techniques, e.g.: (i) it does not require an image of the target fashion item, and (ii) it allows …
AVI-Edit: Audio-sync Video Instance Editing with Granularity-Aware Mask Refiner
School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；State Key Lab of Multimedia Info. Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）；Nat’l Eng. Research Ctr. of Visual Tech., School of Computer Science, Peking University（北京大学计算机学院视觉技术工程研究中心）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
ByteDance Inc（字节跳动公司）；NLPR, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）
The explosion of short videos has dramatically reshaped the manners people socialize, yielding a new trend for daily sharing and access to the latest information. These rich video resources, on the one hand, benefited from the popularization of portable devices with cameras, but on the other, they can not be independent of the valuable editing work contributed by numerous video creators. In this paper, we investigate a novel and practical problem, namely audio beat matching (ABM), which aims to recommend the proper transition time stamps based on the background music. This technique helps to e…
Diffusion models have shown remarkable capabilities in generating high quality and creative images conditioned on text. An interesting application of such models is structure preserving text guided image editing. Existing approaches rely on text conditioned diffusion models such as Stable Diffusion or Imagen and require compute intensive optimization of text embeddings or fine-tuning the model weights for text guided image editing. We explore text guided image editing with a Hybrid Diffusion Model (HDM) architecture similar to DALLE-2. Our architecture consists of a diffusion prior model that …
Generative Adversarial Networks (GANs) are susceptible to bias, learned from either the unbalanced data, or through mode collapse. The networks focus on the core of the data distribution, leaving the tails - or the edges of the distribution - behind. We argue that this bias is responsible not only for fairness concerns, but that it plays a key role in the collapse of latent-traversal editing methods when deviating away from the distribution's core. Building on this observation, we outline a method for mitigating generative bias through a self-conditioning process, where distances in the latent…
University of Science and Technology of China（中国科学技术大学）；Snap Inc（斯奈普公司）；Microsoft Cloud AI（微软云人工智能）；City University of Hong Kong（香港城市大学）
Despite the recent success of face image generation with GANs, conditional hair editing remains challenging due to the under-explored complexity of its geometry and appearance. In this paper, we present MichiGAN (Multi-Input-Conditioned Hair Image GAN), a novel conditional image generation method for interactive portrait hair manipulation. To provide user control over every major hair visual factor, we explicitly disentangle hair into four orthogonal attributes, including shape, structure, appearance, and background. For each of them, we design a corresponding condition module to represent, pr…
We introduce Noise2Music, where a series of diffusion models is trained to generate high-quality 30-second music clips from text prompts. Two types of diffusion models, a generator model, which generates an intermediate representation conditioned on text, and a cascader model, which generates high-fidelity audio conditioned on the intermediate representation and possibly the text, are trained and utilized in succession to generate high-fidelity music. We explore two options for the intermediate representation, one using a spectrogram and the other using audio with lower fidelity. We find that …
Xidian University（西安电子科技大学）；Tencent AI Lab（腾讯人工智能实验室）；Tsinghua University（清华大学）
We present VideoReTalking, a new system to edit the faces of a real-world talking head video according to input audio, producing a high-quality and lip-syncing output video even with a different emotion. Our system disentangles this objective into three sequential tasks: (1) face video generation with a canonical expression; (2) audio-driven lip-sync; and (3) face enhancement for improving photo-realism. Given a talking-head video, we first modify the expression of each frame according to the same expression template using the expression editing network, resulting in a video with the canonical…
From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion
FAIR Team, Meta（Meta FAIR团队）；Universite de Lorraine, CNRS, Inria, LORIA（洛林大学，法国国家科学研究中心，法国国家信息与自动化研究所，洛林计算机科学实验室）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）
City University of Hong Kong（香港城市大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
The research topic of sketch-to-portrait generation has witnessed a boost of progress with deep learning techniques. The recently proposed StyleGAN architectures achieve state-of-the-art generation ability but the original StyleGAN is not friendly for sketch-based creation due to its unconditional generation nature. To address this issue, we propose a direct conditioning strategy to better preserve the spatial information under the StyleGAN framework. Specifically, we introduce Spatially Conditioned StyleGAN (SC-StyleGAN for short), which explicitly injects spatial constraints to the original …
Spatial-Aware Conditioned Fusion for Audio-Visual Navigation
Joint Research Laboratory for Embodied Intelligence, Xinjiang University（新疆大学具身智能联合研究实验室）；Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University（新疆大学丝绸之路多语种认知计算联合国际研究实验室）；School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）
