Yi Yang, Brendan Shillingford, Yannis Assael, Miaosen Wang, Wendi Liu, Yutian Chen, Yu Zhang, Eren Sezener, Luis C. Cobo, Misha Denil, Yusuf Aytar, Nando de Freitas
We describe a system for large-scale audiovisual translation and dubbing, which translates videos from one language to another. The source language's speech content is transcribed to text, translated, and automatically synthesized into target language speech using the original speaker's voice. The visual content is translated by synthesizing lip movements for the speaker to match the translated audio, creating a seamless audiovisual experience in the target language. The audio and visual translation subsystems each contain a large-scale generic synthesis model trained on thousands of hours of data in the corresponding domain. These generic models are fine-tuned to a specific speaker before translation, either using an auxiliary corpus of data from the target speaker, or using the video to be translated itself as the input to the fine-tuning process. This report gives an architectural overview of the full system, as well as an in-depth discussion of the video dubbing component. The role of the audio and text components in relation to the full system is outlined, but their design is not discussed in detail. Translated and dubbed demo videos generated using our system can be viewed at https://www.youtube.com/playlist?list=PLSi232j2ZA6_1Exhof5vndzyfbxAhhEs5
Translators without Borders（无国界译者组织）；Amazon Research（亚马逊研究院）
Dubbing is a type of audiovisual translation where dialogues are translated and enacted so that they give the impression that the media is in the target language. It requires a careful alignment of dubbed recordings with the lip movements of performers in order to achieve visual coherence. In this paper, we deal with the specific problem of prosodic phrase synchronization within the framework of machine dubbing. Our methodology exploits the attention mechanism output in neural machine translation to find plausible phrasing for the translated dialogue lines and then uses them to condition their…
EmoDubber: Towards High Quality and Emotion Controllable Movie Dubbing
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Macquarie University（麦觉里大学）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）；University of Adelaide（阿德莱德大学）
MINT: a Multi-modal Image and Narrative Text Dubbing Dataset for Foley Audio Content Planning and Generation
Beijing University of Posts and Telecommunications（北京邮电大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Tianjin University（天津大学）；Kuaishou Technology Co., Ltd（快手科技有限公司）；Department of Automation, Tsinghua University（清华大学自动化系）；Beijing National Research Center for Information Science and Technology, Tsinghua University（北京信息科学与技术国家研究中心（清华大学））；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Polytechnic University（上海第二工业大学）
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；UT Austin（德克萨斯大学奥斯汀分校）；IBM Research AI（IBM研究院人工智能部门）；Columbia University（哥伦比亚大学）；NYU（纽约大学）
In this paper, we explore self-supervised audio-visual models that learn from instructional videos. Prior work has shown that these models can relate spoken words and sounds to visual content after training on a large-scale dataset of videos, but they were only trained and evaluated on videos in English. To learn multilingual audio-visual representations, we propose a cascaded approach that leverages a model trained on English videos and applies it to audio-visual data in other languages, such as Japanese videos. With our cascaded approach, we show an improvement in retrieval performance of ne…
Face-voice Association in Multilingual Environments (FAME) Challenge 2024 Evaluation Plan
University of Engineering and Technology Taxila（拉合尔工程技术大学塔克西拉分校）；Johannes Kepler University Linz（约翰内斯·开普勒大学林茨分校）；Fortemedia Singapore（Fortemedia新加坡公司）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment
Institute of Image Communication；Network Engineering, Shanghai Key Laboratory of Digital Media Processing；Transmissions, Shanghai Jiao Tong University, Shanghai；School of Communication \& Electronic Engineering, East China Normal University, Shanghai
Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing
Hangzhou Dianzi University（杭州电子科技大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Adelaide（阿德莱德大学）；Macquarie University（麦考瑞大学）
InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing
School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Meituan New Laboratory of Pattern Recognition (NLPR), CASIA（美团模式识别新实验室（NLPR），中国科学院自动化所）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Meituan Division of AMC and Department of ECE, HKUST（美团AMC部门和香港科技大学电子与计算机工程系）；State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（多模态人工智能系统国家重点实验室，中国科学院自动化所）；Meituan School of Artificial Intelligence, University of Chinese Academy of Sciences（美团人工智能学院，中国科学院大学）；Meituan（美团）
SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS
School of Informatics, Xiamen University, China（厦门大学信息学院）；MiLM Plus, Xiaomi Inc., China（小米公司）；School of Electronic Science and Engineering, Xiamen University, China（厦门大学电子科学与技术学院）
Diseño de sonido para producciones audiovisuales e historias sonoras en el aula. Hacia una docencia creativa mediante el uso de herramientas inteligentes
CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China（中国科学院计算技术研究所）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学）；Fudan University, Shanghai, China（复旦大学）；Hangzhou Dianzi University, Hangzhou, China（杭州电子科技大学）；Macquarie University, Sydney, Australia（麦考瑞大学）
FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing
Institute of Computing Technology, Chinese Academy of Sciences \& University of Chinese Academy of Sciences Beijing China；Institute of Computing Technology, Chinese Academy of Sciences Beijing China；Hangzhou Dianzi University Hangzhou China；Macquarie University Sydney Australia；University of Adelaide Adelaide Australia；University of Chinese Academy of Sciences Beijing China；Institute of Computing Technology, Chinese Academy of Sciences \& University of Chinese Academy of Sciences；Institute of Computing Technology, Chinese Academy of Sciences；Hangzhou Dianzi University；Macquarie University；University of Adelaide；University of Chinese Academy of Sciences
Netease Fuxi AI Lab（网易伏羲人工智能实验室）；Xiamen University（厦门大学）；Zhejiang University（浙江大学）
For few-shot learning, it is still a critical challenge to realize photo-realistic face visually dubbing on high-resolution videos. Previous works fail to generate high-fidelity dubbing results. To address the above problem, this paper proposes a Deformation Inpainting Network (DINet) for high-resolution face visually dubbing. Different from previous works relying on multiple up-sample layers to directly generate pixels from latent embeddings, DINet performs spatial deformation on feature maps of reference images to better preserve high-frequency textural details. Specifically, DINet consists …
Shanghai Jiao Tong University（上海交通大学）；University of Electronic Science and Technology of China（电子科技大学）；ByteDance AI Lab（字节跳动人工智能实验室）；Nanyang Technological University（南洋理工大学）；University of Guelph（圭尔夫大学）；Memorial University of Newfoundland（纽芬兰纪念大学）
The task of few-shot visual dubbing focuses on synchronizing the lip movements with arbitrary speech input for any talking head video. Albeit moderate improvements in current approaches, they commonly require high-quality homologous data sources of videos and audios, thus causing the failure to leverage heterogeneous data sufficiently. In practice, it may be intractable to collect the perfect homologous data in some cases, for example, audio-corrupted or picture-blurry videos. To explore this kind of data and support high-fidelity few-shot visual dubbing, in this paper, we novelly propose a si…
Dubbing is a post-production process of re-recording actors' dialogues, which is extensively used in filmmaking and video production. It is usually performed manually by professional voice actors who read lines with proper prosody, and in synchronization with the pre-recorded videos. In this work, we propose Neural Dubber, the first neural network model to solve a novel automatic video dubbing (AVD) task: synthesizing human speech synchronized with the given video from the text. Neural Dubber is a multi-modal text-to-speech (TTS) model that utilizes the lip movement in the video to control the…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Carnegie Mellon University（卡内基梅隆大学）；Istanbul Technical University（伊斯坦布尔理工大学）
In this paper, we propose a neural end-to-end system for voice preserving, lip-synchronous translation of videos. The system is designed to combine multiple component models and produces a video of the original speaker speaking in the target language that is lip-synchronous with the target speech, yet maintains emphases in speech, voice characteristics, face video of the original speaker. The pipeline starts with automatic speech recognition including emphasis detection, followed by a translation model. The translated text is then synthesized by a Text-to-Speech model that recreates the origin…
STSA: Spatial-Temporal Semantic Alignment for Visual Dubbing
University of Science and Technology of China（中国科学技术大学）；Suzhou Institute for Advanced Research, USTC（中国科学技术大学苏州高等研究院）；Harbin Engineering University（哈尔滨工程大学）
We present a method for generating a video of a talking face. The method takes as inputs: (i) still images of the target face, and (ii) an audio speech segment; and outputs a video of the target face lip synched with the audio. The method runs in real time and is applicable to faces and audio not seen at training time. To achieve this we propose an encoder-decoder CNN model that uses a joint embedding of the face and audio to generate synthesised talking face video frames. The model is trained on tens of hours of unlabelled videos. We also show results of re-dubbing videos using speech from a …
Norwegian University of Science and Technology (NTNU)（挪威科技大学）
Smartphones have been employed with biometric-based verification systems to provide security in highly sensitive applications. Audio-visual biometrics are getting popular due to their usability, and also it will be challenging to spoof because of their multimodal nature. In this work, we present an audio-visual smartphone dataset captured in five different recent smartphones. This new dataset contains 103 subjects captured in three different sessions considering the different real-world scenarios. Three different languages are acquired in this dataset to include the problem of language depende…
Technical University of Munich（慕尼黑工业大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Saarland Informatics Campus（萨尔兰信息学园区）
We present Neural Voice Puppetry, a novel approach for audio-driven facial video synthesis. Given an audio sequence of a source person or digital assistant, we generate a photo-realistic output video of a target person that is in sync with the audio of the source input. This audio-driven facial reenactment is driven by a deep neural network that employs a latent 3D face model space. Through the underlying 3D representation, the model inherently learns temporal stability while we leverage neural rendering to generate photo-realistic output frames. Our approach generalizes across different peopl…
University of Bucharest（布加勒斯特大学）；MBZ University of Artificial Intelligence（MBZ人工智能大学）；Linköping University（利德大学）；University of Central Florida（佛罗里达中央大学）
UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions
State Key Laboratory for Novel Software Technology（新型软件技术国家重点实验室）；Tencent Hunyuan（腾讯文英）；Shanghai Jiao Tong University（上海交通大学）；Renmin University of China（中国人民大学）；Tsinghua University（清华大学）；Shanghai AI Lab（上海人工智能实验室）
Istituto Italiano di Tecnologia (IIT)（意大利技术研究院）；University of Insubria（因苏布里亚大学）；University of Engineering and Technology Taxila（塔克西拉工程技术大学）；Information Technology University（信息技术大学）
Recent years have seen a surge in finding association between faces and voices within a cross-modal biometric application along with speaker recognition. Inspired from this, we introduce a challenging task in establishing association between faces and voices across multiple languages spoken by the same set of persons. The aim of this paper is to answer two closely related questions: "Is face-voice association language independent?" and "Can a speaker be recognised irrespective of the spoken language?". These two questions are very important to understand effectiveness and to boost development …
University of Maryland, College Park（马里兰大学帕克分校）；ShareChat
We present 3MASSIV, a multilingual, multimodal and multi-aspect, expertly-annotated dataset of diverse short videos extracted from short-video social media platform - Moj. 3MASSIV comprises of 50k short videos (20 seconds average duration) and 100K unlabeled videos in 11 different languages and captures popular short video trends like pranks, fails, romance, comedy expressed via unique audio-visual formats like self-shot videos, reaction videos, lip-synching, self-sung songs, etc. 3MASSIV presents an opportunity for multimodal and multilingual semantic understanding on these unique videos by a…
University of Sfax（斯法克斯大学）；National School of Engineers（国家工程师学院）
The amount of audio-visual information has increased dramatically with the advent of High Speed Internet. Furthermore, technological advances in recent years in the field of information technology, have simplified the use of video data in various fields by the general public. This made it possible to store large collections of video documents into computer systems. To enable efficient use of these collections, it is necessary to develop tools to facilitate access to these documents and handling them. In this paper we propose a method for indexing and retrieval of video sequences in a video dat…
FREAK: Frequency-modulated High-fidelity and Real-time Audio-driven Talking Portrait Synthesis
School of Computer Science and Engineering, Southeast University（计算机科学与工程学院，东南大学）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China（新一代人工智能技术及其交叉应用重点实验室，教育部，中国）
