Style-Preserving Lip Sync via Audio-Aware Style Reference
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Shandong Inspur Database Technology Co., Ltd.（山东 Inspur 数据库技术有限公司）；Peking University（北京大学）；Peng Cheng Laboratory（鹏城实验室）
Weizhi Zhong, Jichang Li, Yinqi Cai, Ming Li, Feng Gao, Liang Lin, Guanbin Li
Audio-driven lip sync has recently drawn significant attention due to its widespread application in the multimedia domain. Individuals exhibit distinct lip shapes when speaking the same utterance, attributed to the unique speaking styles of individuals, posing a notable challenge for audio-driven lip sync. Earlier methods for such task often bypassed the modeling of personalized speaking styles, resulting in sub-optimal lip sync conforming to the general styles. Recent lip sync techniques attempt to guide the lip sync for arbitrary audio by aggregating information from a style reference video, yet they can not preserve the speaking styles well due to their inaccuracy in style aggregation. This work proposes an innovative audio-aware style reference scheme that effectively leverages the relationships between input audio and reference audio from style reference video to address the style-preserving audio-driven lip sync. Specifically, we first develop an advanced Transformer-based model adept at predicting lip motion corresponding to the input audio, augmented by the style information aggregated through cross-attention layers from style reference video. Afterwards, to better render the lip motion into realistic talking face video, we devise a conditional latent diffusion model, integrating lip motion through modulated convolutional layers and fusing reference facial images via spatial cross-attention layers. Extensive experiments validate the efficacy of the proposed approach in achieving precise lip sync, preserving speaking styles, and generating high-fidelity, realistic talking face videos.
The challenge of talking face generation from speech lies in aligning two different modal information, audio and video, such that the mouth region corresponds to input audio. Previous methods either exploit audio-visual representation learning or leverage intermediate structural information such as landmarks and 3D models. However, they struggle to synthesize fine details of the lips varying at the phoneme level as they do not sufficiently provide visual information of the lips at the video synthesis step. To overcome this limitation, our work proposes Audio-Lip Memory that brings in visual in…
SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion
University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）；Nanyang Technological University（南洋理工大学）；City University of Hong Kong（香港城市大学）
Xidian University（西安电子科技大学）；Tencent AI Lab（腾讯人工智能实验室）；Tsinghua University（清华大学）
We present VideoReTalking, a new system to edit the faces of a real-world talking head video according to input audio, producing a high-quality and lip-syncing output video even with a different emotion. Our system disentangles this objective into three sequential tasks: (1) face video generation with a canonical expression; (2) audio-driven lip-sync; and (3) face enhancement for improving photo-realism. Given a talking-head video, we first modify the expression of each frame according to the same expression template using the expression editing network, resulting in a video with the canonical…
Audio-Sync Video Generation with Multi-Stream Temporal Control
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Nat’l Key Lab of General AI, School of Intelligence Science and Technology, Peking University（国家通用人工智能实验室，北京大学智能科学与技术学院）；Nat’l Eng. Research Ctr. of Visual Tech., School of Computer Science, Peking University（国家视觉技术工程研究中心，北京大学计算机学院）
本文提出 MTV 框架，通过分离语音、音效和音乐多流音频实现细粒度音视频同步控制，并构建 DEMIX 数据集，在六项指标上达到最先进性能。
JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization
Zhejiang University（浙江大学）；National University of Singapore（新加坡国立大学）；University of Science and Technology of China（中国科学技术大学）；University of Rochester（罗切斯特大学）；Nanyang Technological University（南洋理工大学）
AVI-Edit: Audio-sync Video Instance Editing with Granularity-Aware Mask Refiner
School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；State Key Lab of Multimedia Info. Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）；Nat’l Eng. Research Ctr. of Visual Tech., School of Computer Science, Peking University（北京大学计算机学院视觉技术工程研究中心）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
The University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Deep learning based visual to sound generation systems essentially need to be developed particularly considering the synchronicity aspects of visual and audio features with time. In this research we introduce a novel task of guiding a class conditioned generative adversarial network with the temporal visual information of a video input for visual to sound generation task adapting the synchronicity traits between audio-visual modalities. Our proposed FoleyGAN model is capable of conditioning action sequences of visual events leading towards generating visually aligned realistic sound tracks. We…
Thapar Institute of Engineering and Technology（塔帕尔工程技术学院）；Indian Institute of Technology Ropar（印度理工学院鲁尔基分校）；Oxford Brookes University（牛津布鲁克斯大学）
Recently, there has been numerous breakthroughs in face hallucination tasks. However, the task remains rather challenging in videos in comparison to the images due to inherent consistency issues. The presence of extra temporal dimension in video face hallucination makes it non-trivial to learn the facial motion through out the sequence. In order to learn these fine spatio-temporal motion details, we propose a novel cross-modal audio-visual Video Face Hallucination Generative Adversarial Network (VFH-GAN). The architecture exploits the semantic correlation of between the movement of the facial …
VABench: A Comprehensive Benchmark for Audio-Video Generation
Peking University（北京大学）；Zhongguancun Academy（中关村学院）；Beijing Key Laboratory of Data Intelligence (Peking University)（北京市数据智能重点实验室（北京大学））；Ant Group（蚂蚁集团）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Huazhong University of Science and Technology（华中科技大学）
Assessing Identity Leakage in Talking Face Generation: Metrics and Evaluation Framework
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；KIT Campus Transfer GmbH (KCT)（KIT校园转移公司）；Istanbul Technical University（伊斯坦布尔技术大学）；Carnegie Mellon University（卡内基梅隆大学）
Deep Innovation R&D Center, Moviebook（墨迹天气深创新研发中心）
Lip sync has emerged as a promising technique for generating mouth movements from audio signals. However, synthesizing a high-resolution and photorealistic virtual news anchor is still challenging. Lack of natural appearance, visual consistency, and processing efficiency are the main problems with existing methods. This paper presents a novel lip-sync framework specially designed for producing high-fidelity virtual news anchors. A pair of Temporal Convolutional Networks are used to learn the cross-modal sequential mapping from audio signals to mouth movements, followed by a neural rendering ne…
Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animation
Beihang University（北京航空航天大学）；Peng Cheng Laboratory（鹏城实验室）；North China University of Technology（华北理工大学）；University of Washington（华盛顿大学）；University of Copenhagen（哥本哈根大学）
While recent advances in deep neural networks have made it possible to render high-quality images, generating photo-realistic and personalized talking head remains challenging. With given audio, the key to tackling this task is synchronizing lip movement and simultaneously generating personalized attributes like head movement and eye blink. In this work, we observe that the input audio is highly correlated to lip motion while less correlated to other personalized attributes (e.g., head movements). Inspired by this, we propose a novel framework based on neural radiance field to pursue high-fide…
Recognizing speech from silent lip movement, which is called lip reading, is a challenging task due to 1) the inherent information insufficiency of lip movement to fully represent the speech, and 2) the existence of homophenes that have similar lip movement with different pronunciations. In this paper, we try to alleviate the aforementioned two challenges in lip reading by proposing a Multi-head Visual-audio Memory (MVM). Firstly, MVM is trained with audio-visual datasets and remembers audio representations by modelling the inter-relationships of paired audio-visual representations. At the inf…
Tianjin University（天津大学）；National Institute of Informatics（日本国立信息学研究所）
Audio-visual (AV) lip biometrics is a promising authentication technique that leverages the benefits of both the audio and visual modalities in speech communication. Previous works have demonstrated the usefulness of AV lip biometrics. However, the lack of a sizeable AV database hinders the exploration of deep-learning-based audio-visual lip biometrics. To address this problem, we compile a moderate-size database using existing public databases. Meanwhile, we establish the DeepLip AV lip biometrics system realized with a convolutional neural network (CNN) based video module, a time-delay neura…
PC-Talk: Precise Facial Animation Control for Audio-Driven Talking Face Generation
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS部）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Psyche AI.INC（Psyche AI公司）；HKUST（香港科技大学）；CAIR, HKISI, Chinese Academy of Sciences（中国科学院计算智能研究所）；SCSE, FIE, M.U.S.T（M.U.S.T的SCSE、FIE部门）
Tokyo Institute of Technology（东京工业大学）；Baidu Inc.（百度公司）；The University of Sydney（悉尼大学）；Monash University（莫纳什大学）；Nanyang Technological University（南洋理工大学）
Previous studies have explored generating accurately lip-synced talking faces for arbitrary targets given audio conditions. However, most of them deform or generate the whole facial area, leading to non-realistic results. In this work, we delve into the formulation of altering only the mouth shapes of the target person. This requires masking a large percentage of the original image and seamlessly inpainting it with the aid of audio and reference frames. To this end, we propose the Audio-Visual Context-Aware Transformer (AV-CAT) framework, which produces accurate lip-sync with photo-realistic q…
Talking face generation with great practical significance has attracted more attention in recent audio-visual studies. How to achieve accurate lip synchronization is a long-standing challenge to be further investigated. Motivated by xxx, in this paper, an AttnWav2Lip model is proposed by incorporating spatial attention module and channel attention module into lip-syncing strategy. Rather than focusing on the unimportant regions of the face image, the proposed AttnWav2Lip model is able to pay more attention on the lip region reconstruction. To our limited knowledge, this is the first attempt to…
Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from Lip Reading Expert
Grad. School of Artificial Intelligence（人工智能研究生院）；POSTECH（浦项科技大学）；ENSC, Bordeaux INP, France（法国波尔多国立高等工程学院）；KRAFTON, Korea（韩国KRAFTON公司）；Inst. for Convergence Research and Education in Advanced Technology, Yonsei University, Korea（延世大学先进技术融合研究与教育学院）
In this paper, we address the problem of lip-voice synchronisation in videos containing human face and voice. Our approach is based on determining if the lips motion and the voice in a video are synchronised or not, depending on their audio-visual correspondence score. We propose an audio-visual cross-modal transformer-based model that outperforms several baseline models in the audio-visual synchronisation task on the standard lip-reading speech benchmark dataset LRS2. While the existing methods focus mainly on lip synchronisation in speech videos, we also consider the special case of the sing…
The American University in Cairo（开罗美国大学）；Nahdet Misr AI（纳赫代特米斯尔人工智能公司）
Despite the advancement in the domain of audio and audio-visual speech recognition, visual speech recognition systems are still quite under-explored due to the visual ambiguity of some phonemes. In this work, we propose a new lip-reading model that combines three contributions. First, the model front-end adopts a spatio-temporal attention mechanism to help extract the informative data from the input visual frames. Second, the model back-end utilizes a sequence-level and frame-level Knowledge Distillation (KD) techniques that allow leveraging audio data during the visual model training. Third, …
The Hong Kong University of Science and Technology（香港科技大学）
Most lip-to-speech (LTS) synthesis models are trained and evaluated under the assumption that the audio-video pairs in the dataset are perfectly synchronized. In this work, we show that the commonly used audio-visual datasets, such as GRID, TCD-TIMIT, and Lip2Wav, can have data asynchrony issues. Training lip-to-speech with such datasets may further cause the model asynchrony issue -- that is, the generated speech and the input video are out of sync. To address these asynchrony issues, we propose a synchronized lip-to-speech (SLTS) model with an automatic synchronization mechanism (ASM) to cor…
Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）；Meta AI
This paper investigates self-supervised pre-training for audio-visual speaker representation learning where a visual stream showing the speaker's mouth area is used alongside speech as inputs. Our study focuses on the Audio-Visual Hidden Unit BERT (AV-HuBERT) approach, a recently developed general-purpose audio-visual speech pre-training framework. We conducted extensive experiments probing the effectiveness of pre-training and visual modality. Experimental results suggest that AV-HuBERT generalizes decently to speaker related downstream tasks, improving label efficiency by roughly ten fold fo…
