面向跨语言语音合成的零样本情感迁移
Zero-Shot Emotion Transfer For Cross-Lingual Speech Synthesis
AI总结：
针对跨语言零样本情感迁移语音合成的口音与情感建模难题，基于DelightfulTTS架构分别用NPC模块建模语言特定韵律、从HuBERT提取共通情感并结合分层建模，实现了无情感训练数据下的单语说话人双语情感语音合成。
摘要：
跨语言语音合成中的零样本情感迁移旨在将源语言任意参考语音中的情感迁移到目标语言的合成语音中。构建这类系统面临着外来口音不自然、难以建模不同语言共通情感表达的挑战。本文基于DelightfulTTS神经架构，通过引入专门设计的模块分别建模语言特定韵律特征与语言共通情感表达，以应对上述挑战。具体而言，通过非自回归预测编码（NPC）模块学习语言特定的语音韵律，以提升跨语言合成语音的自然度；从泛化能力较强的预训练自监督模型HuBERT中提取不同语言间的共通情感表达，还进一步采用分层情感建模来捕捉不同语言间更全面的情感信息。实验结果表明，所提框架可在目标单语说话人无情感训练数据的情况下，有效合成双语情感语音。
英文摘要：
Zero-shot emotion transfer in cross-lingual speech synthesis aims to transfer emotion from an arbitrary speech reference in the source language to the synthetic speech in the target language. Building such a system faces challenges of unnatural foreign accents and difficulty in modeling the shared emotional expressions of different languages. Building on the DelightfulTTS neural architecture, this paper addresses these challenges by introducing specifically-designed modules to model the language-specific prosody features and language-shared emotional expressions separately. Specifically, the language-specific speech prosody is learned by a non-autoregressive predictive coding (NPC) module to improve the naturalness of the synthetic cross-lingual speech. The shared emotional expression between different languages is extracted from a pre-trained self-supervised model HuBERT with strong generalization capabilities. We further use hierarchical emotion modeling to capture more comprehensive emotions across different languages. Experimental results demonstrate the proposed framework's effectiveness in synthesizing bi-lingual emotional speech for the monolingual target speaker without emotional training data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
