Audio-visual alignment after dubbing is a challenging research problem. To this end, we propose a novel method, DubWise Multi-modal Large Language Model (LLM)-based Text-to-Speech (TTS), which can control the speech duration of synthesized speech in such a way that it aligns well with the speakers lip movements given in the reference video even when the spoken text is different or in a different language. To accomplish this, we propose to utilize cross-modal attention techniques in a pre-trained GPT-based TTS. We combine linguistic tokens from text, speaker identity tokens via a voice cloning network, and video tokens via a proposed duration controller network. We demonstrate the effectiveness of our system on the Lip2Wav-Chemistry and LRS2 datasets. Also, the proposed method achieves improved lip sync and naturalness compared to the SOTAs for the same language but different text (i.e., non-parallel) and the different language, different text (i.e., cross-lingual) scenarios.
Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimization
Alibaba Digital Media and Entertainment Group（阿里巴巴数字媒体与娱乐集团）；School of Software Engineering, Huazhong University of Science and Technology（华中科技大学软件学院）；Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University（北京交通大学交通数据挖掘与具身智能重点实验室）
Learning to Dub Movies via Hierarchical Prosody Models
Shandong University（山东大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Lishui Institute of Hangzhou Dianzi University（杭州电子科技大学丽水研究院）；Australian Institute for Machine Learning, University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；University of Science and Technology of China（中国科学技术大学）；University of California（加利福尼亚大学）；Yonsei University（延世大学）
PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbing
Dept. of AI Convergence（人工智能融合系）；Dept of Electrical Engineering and Computer Science（电子工程与计算机科学系）；Gwangju Institute of Science and Technology（全州科学技术院）；AunionAI Co., Ltd.（AunionAI公司）
EmoDubber: Towards High Quality and Emotion Controllable Movie Dubbing
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Macquarie University（麦觉里大学）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）；University of Adelaide（阿德莱德大学）
WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS
South China University of Technology（南方科技大学）；Huya Inc.（Huya公司）；Tongji University（同济大学）；The Hongkong polytechnic university（香港理工大学）；Foshan University（佛山大学）
Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters
AWS AI Labs（AWS人工智能实验室）；School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；Center for Information and Language Processing, LMU Munich（慕尼黑大学信息与语言处理中心）
Controllable Accent Normalization via Discrete Diffusion
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanjing University（南京大学）；Tencent Ethereal Audio Lab（腾讯虚音频实验室）；Shenzhen Loop Area Institute（深圳河套学院）
TokAN: Accent Normalization Using Self-Supervised Speech Tokens
School of Data Science (SDS), The Chinese University of Hong Kong, Shenzhen (CUHKSZ)（香港中文大学（深圳）数据科学学院）；Tencent Ethereal Audio Lab, Tencent（腾讯虚幻音频实验室）；School of Intelligence Science and Technology, Nanjing University, Suzhou（南京大学苏州校区智能科学与技术学院）；Shenzhen Loop Area Institute（深圳河套学院）；X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学与工程学院X-LANCE实验室）；School of Artificial Intelligence (SAI), CUHKSZ（香港中文大学（深圳）人工智能学院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
