arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.17005eess.AScs.CLcs.SD
收藏

具有精确对齐的高质量自动配音:通过自监督离散语音单元进行监督

High-Quality Automatic Voice Over with Accurate Alignment: Supervision through Self-Supervised Discrete Speech Units

National University of Singapore(新加坡国立大学);The University of Texas at Dallas(德克萨斯大学达拉斯分校);Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

Junchen Lu, Berrak Sisman, Mingyang Zhang, Haizhou Li

AI总结:

本文提出一种基于自监督离散语音单元预测的自动配音方法,通过提供更直接的对齐监督并缓解上下文与声学特征的不匹配,显著提升了唇音同步与语音质量。

摘要:

自动配音(AVO)的目标是根据文本脚本生成与无声视频同步的语音。基于文本到语音合成(TTS)构建的近期AVO框架已展现出令人印象深刻的效果。然而,当前AVO的声学特征重建学习目标为跨模态对齐学习带来了间接监督,从而限制了同步性能与合成语音质量。为此,我们提出一种新颖的AVO方法,利用自监督离散语音单元预测的学习目标,这不仅为对齐学习提供了更直接的监督,还缓解了文本-视频上下文与声学特征之间的不匹配。实验结果表明,所提方法在客观与主观评估中均优于基线,实现了显著的唇音同步与高语音质量。代码和语音样本已公开。

英文摘要:

The goal of Automatic Voice Over (AVO) is to generate speech in sync with a silent video given its text script. Recent AVO frameworks built upon text-to-speech synthesis (TTS) have shown impressive results. However, the current AVO learning objective of acoustic feature reconstruction brings in indirect supervision for inter-modal alignment learning, thus limiting the synchronization performance and synthetic speech quality. To this end, we propose a novel AVO method leveraging the learning objective of self-supervised discrete speech unit prediction, which not only provides more direct supervision for the alignment learning, but also alleviates the mismatch between the text-video context and acoustic features. Experimental results show that our proposed method achieves remarkable lip-speech synchronization and high speech quality by outperforming baselines in both objective and subjective evaluations. Code and speech samples are publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑