AffectEcho：面向语音合成的说话人无关且语言无关的情感与情绪迁移
AffectEcho: Speaker Independent and Language-Agnostic Emotion and Affect Transfer for Speech Synthesis
AI总结：
本文提出 AffectEcho，利用五级情绪强度的 VQ 码本隐式建模量化情绪嵌入，在双语语料上实现说话人无关、语言无关的语音情绪迁移，并取得最优定性与定量结果。
摘要：
情感（affect）是一种涵盖效价、唤醒度和强度的情绪特征，也是实现真实对话的关键属性。现有文本转语音（TTS）和语音转语音系统依赖强度嵌入向量与全局风格令牌来捕捉情绪，但这些模型将情绪表示为风格的一个组成部分，或以离散类别表示情绪。我们提出 AffectEcho，这是一种情绪翻译模型，使用 Vector Quantized 码本在具有五级情绪强度的量化空间中对情绪建模，以捕捉同一情绪中的复杂细微差别和微妙差异。量化情绪嵌入是从口语音频样本中隐式获得的，无需 one-hot 向量或显式强度嵌入。实验结果表明，我们的方法能够有效控制生成语音的情绪，同时保留每位说话人独特的身份、风格和情感节奏。我们利用从英语和中文双语音频语料库中学到的量化情绪嵌入，通过从参考语音到目标语音的情绪迁移任务，展示了其语言无关的情绪建模能力。我们在定性和定量指标上均取得了 state-of-art 结果。
英文摘要：
Affect is an emotional characteristic encompassing valence, arousal, and intensity, and is a crucial attribute for enabling authentic conversations. While existing text-to-speech (TTS) and speech-to-speech systems rely on strength embedding vectors and global style tokens to capture emotions, these models represent emotions as a component of style or represent them in discrete categories. We propose AffectEcho, an emotion translation model, that uses a Vector Quantized codebook to model emotions within a quantized space featuring five levels of affect intensity to capture complex nuances and subtle differences in the same emotion. The quantized emotional embeddings are implicitly derived from spoken speech samples, eliminating the need for one-hot vectors or explicit strength embeddings. Experimental results demonstrate the effectiveness of our approach in controlling the emotions of generated speech while preserving identity, style, and emotional cadence unique to each speaker. We showcase the language-independent emotion modeling capability of the quantized emotional embeddings learned from a bilingual (English and Chinese) speech corpus with an emotion transfer task from a reference speech to a target speech. We achieve state-of-art results on both qualitative and quantitative metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
