StyleDubber：面向电影配音的多尺度风格学习
StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing
AI总结：
本文提出StyleDubber，通过音素级多模态风格适配、话语级风格学习和音素引导唇形对齐，解决电影配音中帧切分导致的发音不完整与身份不稳问题。
摘要：
给定剧本，电影配音（视觉语音克隆，V2C）的挑战在于根据参考音轨的语调，生成在时间和情感上都与视频良好对齐的语音。现有最先进的V2C模型按照视频帧之间的分界来切分剧本中的音素，这解决了时间对齐问题，但会导致音素发音不完整以及身份稳定性较差。为解决该问题，我们提出了StyleDubber，它将配音学习从帧级切换到音素级。该模型包含三个主要组件：（1）在音素级运行的多模态风格适配器，用于从参考音频中学习发音风格，并根据视频呈现的面部情感生成中间表示；（2）话语级风格学习模块，从中间嵌入出发指导梅尔频谱图解码和精炼过程，以提升整体风格表达；以及（3）用于保持唇形同步的音素引导唇形对齐器。在两个主要基准V2C和Grid上进行的大量实验表明，与当前最先进方法相比，所提方法表现出良好性能。代码将在https://github.com/GalaxyCong/StyleDubber公开。
英文摘要：
Given a script, the challenge in Movie Dubbing (Visual Voice Cloning, V2C) is to generate speech that aligns well with the video in both time and emotion, based on the tone of a reference audio track. Existing state-of-the-art V2C models break the phonemes in the script according to the divisions between video frames, which solves the temporal alignment problem but leads to incomplete phoneme pronunciation and poor identity stability. To address this problem, we propose StyleDubber, which switches dubbing learning from the frame level to phoneme level. It contains three main components: (1) A multimodal style adaptor operating at the phoneme level to learn pronunciation style from the reference audio, and generate intermediate representations informed by the facial emotion presented in the video; (2) An utterance-level style learning module, which guides both the mel-spectrogram decoding and the refining processes from the intermediate embeddings to improve the overall style expression; And (3) a phoneme-guided lip aligner to maintain lip sync. Extensive experiments on two of the primary benchmarks, V2C and Grid, demonstrate the favorable performance of the proposed method as compared to the current stateof-the-art. The code will be made available at https://github.com/GalaxyCong/StyleDubber.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
