面向自动配音的联合多尺度跨语言说话风格迁移与双向注意力机制
Joint Multi-scale Cross-lingual Speaking Style Transfer with Bidirectional Attention Mechanism for Automatic Dubbing
AI总结：
本文提出联合多尺度跨语言说话风格迁移框架，通过双向注意力机制在语句级和词级同时建模双向风格迁移，并利用风格增强的FastSpeech 2合成语音，在自动配音中超越仅迁移时长的基线系统。
摘要：
自动配音能够生成输入语音在另一种语言中的对应版本，可广泛应用于视频和游戏本地化等许多现实场景。除了合成翻译后的脚本外，自动配音还需要进一步将原始语言中的说话风格迁移到配音语音中，从而给观众留下角色正在使用其母语说话的印象。然而，当前最先进的自动配音系统仅对时长和语速的迁移进行建模，忽略了情感、语调和重音等说话风格中的其他方面，而这些方面对于完整塑造角色和理解语音同样至关重要。本文提出了一种联合多尺度跨语言说话风格迁移框架，同时在全局（即语句级）和局部（即词级）两个尺度上对语言之间的双向说话风格迁移进行建模。该框架提取每种语言中的全局和局部说话风格，并利用每个方向上的编码器-解码器框架以及两个方向共享的双向注意力机制，预测另一种语言中的全局和局部说话风格。随后，利用多尺度说话风格增强的FastSpeech 2将预测得到的全局和局部说话风格合成为每种语言的语音。实验结果表明，所提框架是有效的，在客观和主观评估中均优于仅进行时长迁移的基线系统。
英文摘要：
Automatic dubbing, which generates a corresponding version of the input speech in another language, could be widely utilized in many real-world scenarios such as video and game localization. In addition to synthesizing the translated scripts, automatic dubbing needs to further transfer the speaking style in the original language to the dubbed speeches to give audiences the impression that the characters are speaking in their native tongue. However, state-of-the-art automatic dubbing systems only model the transfer on duration and speaking rate, neglecting the other aspects in speaking style such as emotion, intonation and emphasis which are also crucial to fully perform the characters and speech understanding. In this paper, we propose a joint multi-scale cross-lingual speaking style transfer framework to simultaneously model the bidirectional speaking style transfer between languages at both global (i.e. utterance level) and local (i.e. word level) scales. The global and local speaking styles in each language are extracted and utilized to predicted the global and local speaking styles in the other language with an encoder-decoder framework for each direction and a shared bidirectional attention mechanism for both directions. A multi-scale speaking style enhanced FastSpeech 2 is then utilized to synthesize the predicted the global and local speaking styles to speech for each language. Experiment results demonstrate the effectiveness of our proposed framework, which outperforms a baseline with only duration transfer in both objective and subjective evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
