Video dubbing aims to synthesize realistic, lip-synced videos from a reference video and a driving audio signal. Although existing methods can accurately generate mouth shapes driven by audio, they often fail to preserve identity-specific features, largely because they do not effectively capture the nuanced interplay between audio cues and the visual attributes of reference identity . As a result, the generated outputs frequently lack fidelity in reproducing the unique textural and structural details of the reference identity. To address these limitations, we propose IPTalker, a novel and robust framework for video dubbing that achieves seamless alignment between driving audio and reference identity while ensuring both lip-sync accuracy and high-fidelity identity preservation. At the core of IPTalker is a transformer-based alignment mechanism designed to dynamically capture and model the correspondence between audio features and reference images, thereby enabling precise, identity-aware audio-visual integration. Building on this alignment, a motion warping strategy further refines the results by spatially deforming reference images to match the target audio-driven configuration. A dedicated refinement process then mitigates occlusion artifacts and enhances the preservation of fine-grained textures, such as mouth details and skin features. Extensive qualitative and quantitative evaluations demonstrate that IPTalker consistently outperforms existing approaches in terms of realism, lip synchronization, and identity retention, establishing a new state of the art for high-quality, identity-consistent video dubbing.
InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing
School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Meituan New Laboratory of Pattern Recognition (NLPR), CASIA（美团模式识别新实验室（NLPR），中国科学院自动化所）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Meituan Division of AMC and Department of ECE, HKUST（美团AMC部门和香港科技大学电子与计算机工程系）；State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（多模态人工智能系统国家重点实验室，中国科学院自动化所）；Meituan School of Artificial Intelligence, University of Chinese Academy of Sciences（美团人工智能学院，中国科学院大学）；Meituan（美团）
Shanghai Jiao Tong University（上海交通大学）；University of Electronic Science and Technology of China（电子科技大学）；ByteDance AI Lab（字节跳动人工智能实验室）；Nanyang Technological University（南洋理工大学）；University of Guelph（圭尔夫大学）；Memorial University of Newfoundland（纽芬兰纪念大学）
The task of few-shot visual dubbing focuses on synchronizing the lip movements with arbitrary speech input for any talking head video. Albeit moderate improvements in current approaches, they commonly require high-quality homologous data sources of videos and audios, thus causing the failure to leverage heterogeneous data sufficiently. In practice, it may be intractable to collect the perfect homologous data in some cases, for example, audio-corrupted or picture-blurry videos. To explore this kind of data and support high-fidelity few-shot visual dubbing, in this paper, we novelly propose a si…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Stanford University（斯坦福大学）；DisneyResearch|Studios（迪士尼研究工作室）；University of Bath（巴斯大学）
Dubbing is a technique for translating video content from one language to another. However, state-of-the-art visual dubbing techniques directly copy facial expressions from source to target actors without considering identity-specific idiosyncrasies such as a unique type of smile. We present a style-preserving visual dubbing approach from single video inputs, which maintains the signature style of target actors when modifying facial expressions, including mouth motions, to match foreign languages. At the heart of our approach is the concept of motion style, in particular for facial expressions…
CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China（中国科学院计算技术研究所）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学）；Fudan University, Shanghai, China（复旦大学）；Hangzhou Dianzi University, Hangzhou, China（杭州电子科技大学）；Macquarie University, Sydney, Australia（麦考瑞大学）
ShanghaiTech University（上海科技大学）；Chinese Academy of Sciences（中国科学院）；Shanghai Institute of Microsystem and Information Technology（上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
We tackle human image synthesis, including human motion imitation, appearance transfer, and novel view synthesis, within a unified framework. It means that the model, once being trained, can be used to handle all these tasks. The existing task-specific methods mainly use 2D keypoints to estimate the human body structure. However, they only express the position information with no abilities to characterize the personalized shape of the person and model the limb rotations. In this paper, we propose to use a 3D body mesh recovery module to disentangle the pose and shape. It can not only model the…
Face animation aims at creating photo-realistic portrait videos with animated poses and expressions. A common practice is to generate displacement fields that are used to warp pixels and features from source to target. However, prior attempts often produce sub-optimal displacements. In this work, we present a geometry driven model and propose two geometric patterns as guidance: 3D face rendered displacement maps and posed neural codes. The model can optionally use one of the patterns as guidance for displacement estimation. To model displacements at locations not covered by the face model (e.g…
FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing
Institute of Computing Technology, Chinese Academy of Sciences \& University of Chinese Academy of Sciences Beijing China；Institute of Computing Technology, Chinese Academy of Sciences Beijing China；Hangzhou Dianzi University Hangzhou China；Macquarie University Sydney Australia；University of Adelaide Adelaide Australia；University of Chinese Academy of Sciences Beijing China；Institute of Computing Technology, Chinese Academy of Sciences \& University of Chinese Academy of Sciences；Institute of Computing Technology, Chinese Academy of Sciences；Hangzhou Dianzi University；Macquarie University；University of Adelaide；University of Chinese Academy of Sciences
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
One-shot face re-enactment is a challenging task due to the identity mismatch between source and driving faces. Specifically, the suboptimally disentangled identity information of driving subjects would inevitably interfere with the re-enactment results and lead to face shape distortion. To solve this problem, this paper proposes to use 3D Morphable Model (3DMM) for explicit facial semantic decomposition and identity disentanglement. Instead of using 3D coefficients alone for re-enactment control, we take the advantage of the generative ability of 3DMM to render textured face proxies. These pr…
Navigating Large-Pose Challenge for High-Fidelity Face Reenactment with Video Diffusion Model
National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu, 610065, China（合成视觉基础科学国家重点实验室，四川大学，成都，610065，中国）；School of Cyber Science and Engineering, Sichuan University, Chengdu, 610065, China（网络科学与工程学院，四川大学，成都，610065，中国）；College of Computer Science, Sichuan University, Chengdu, 610065, China（计算机科学学院，四川大学，成都，610065，中国）
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Computer Science & Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）
Given an input face photo, the goal of caricature generation is to produce stylized, exaggerated caricatures that share the same identity as the photo. It requires simultaneous style transfer and shape exaggeration with rich diversity, and meanwhile preserving the identity of the input. To address this challenging problem, we propose a novel framework called Multi-Warping GAN (MW-GAN), including a style network and a geometric network that are designed to conduct style transfer and geometric exaggeration respectively. We bridge the gap between the style and landmarks of an image with correspon…
ShanghaiTech University（上海科技大学）；Tencent AI Lab（腾讯人工智能实验室）
We tackle the human motion imitation, appearance transfer, and novel view synthesis within a unified framework, which means that the model once being trained can be used to handle all these tasks. The existing task-specific methods mainly use 2D keypoints (pose) to estimate the human body structure. However, they only expresses the position information with no abilities to characterize the personalized shape of the individual person and model the limbs rotations. In this paper, we propose to use a 3D body mesh recovery module to disentangle the pose and shape, which can not only model the join…
When there is a mismatch between the target identity and the driver identity, face reenactment suffers severe degradation in the quality of the result, especially in a few-shot setting. The identity preservation problem, where the model loses the detailed information of the target leading to a defective output, is the most common failure mode. The problem has several potential sources such as the identity of the driver leaking due to the identity mismatch, or dealing with unseen large poses. To overcome such problems, we introduce components that address the mentioned problem: image attention …
FLOW: Feature-Level Optimal Warping for Generalized Remote Physiological Measurement
Great Bay University（大湾区大学）；Hefei University of Technology（合肥工业大学）；Southern University of Science and Technology（南方科技大学）；Macao Polytechnic University（澳门理工大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Dongguan Key Laboratory for Intelligence and Information Technology（东莞市智能与信息技术重点实验室）
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；PhiGent Robotics（鉴智机器人）
Talking head synthesis is an emerging technology with wide applications in film dubbing, virtual avatars and online education. Recent NeRF-based methods generate more natural talking videos, as they better capture the 3D structural information of faces. However, a specific model needs to be trained for each identity with a large dataset. In this paper, we propose Dynamic Facial Radiance Fields (DFRF) for few-shot talking head synthesis, which can rapidly generalize to an unseen identity with few training data. Different from the existing NeRF-based methods which directly encode the 3D geometry…
We propose an attention-based networks for transferring motions between arbitrary objects. Given a source image(s) and a driving video, our networks animate the subject in the source images according to the motion in the driving video. In our attention mechanism, dense similarities between the learned keypoints in the source and the driving images are computed in order to retrieve the appearance information from the source images. Taking a different approach from the well-studied warping based models, our attention-based model has several advantages. By reassembling non-locally searched pieces…
Editing faces in videos is a popular yet challenging aspect of computer vision and graphics, which encompasses several applications including facial attractiveness enhancement, makeup transfer, face replacement, and expression manipulation. Simply applying image-based warping algorithms to video-based face editing produces temporal incoherence in the synthesized videos because it is impossible to consistently localize facial features in two frames representing two different faces in two different videos (or even two consecutive frames representing the same face in one video). Therefore, high p…
Google, Inc.（谷歌公司）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
We present a method for synthesizing a frontal, neutral-expression image of a person's face given an input face photograph. This is achieved by learning to generate facial landmarks and textures from features extracted from a facial-recognition network. Unlike previous approaches, our encoding feature vector is largely invariant to lighting, pose, and facial expression. Exploiting this invariance, we train our decoder network using only frontal, neutral-expression photographs. Since these photographs are well aligned, we can decompose them into a sparse set of landmark points and aligned textu…
Zhejiang University（浙江大学）；University of Bath（巴斯大学）
Sharing short personalized videos to various social media networks has become quite popular in recent years. This raises the need for digital retouching of portraits in videos. However, applying portrait image editing directly on portrait video frames cannot generate smooth and stable video sequences. To this end, we present a robust and easy-to-use parametric method to reshape the portrait in a video to produce smooth retouched results. Given an input portrait video, our method consists of two main stages: stabilized face reconstruction, and continuous video reshaping. In the first stage, we …
Temporally-Aligned Evaluation for Audio-Driven Talking Head Generation
School of Business, University of New South Wales (UNSW)（新南威尔士大学商学院）；School of Engineering and Built Environment, Griffith University（格里菲斯大学工程与环境学院）；Data61/CSIRO（Data61/澳大利亚国家科学委员会）
