arXiv 2510.111242025-10-13cs.SD

跨语言情感TTS的扰动自监督表示：情感与说话人的分阶段建模

Perturbation Self-Supervised Representations for Cross-Lingual Emotion TTS: Stage-Wise Modeling of Emotion and Speaker

Tianjin Key Laboratory of Cognitive Computing and Application（天津认知计算与应用重点实验室）；College of Intelligence and Computing（智能计算学院）；Tianjin University（天津大学）；Institute of Artificial Intelligence (TeleAI)（人工智能研究所）；China Telecom, China（中国电信）；Duke Kunshan University（杜克大学昆山学院）；Northwestern Polytechnical University（西北工业大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究所，中国科学院）

Cheng Gong, Chunyu Qiang, Tianrui Wang, Yu Jiang, Yuheng Lu, Ruihao Jing, Xiaoxiao Miao, Xiaolei Zhang, Longbiao Wang, Jianwu Dang

备注：: Submitted to Expert Systems with Applications,11 pages

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出基于扰动自监督表示的两阶段跨语言情感TTS框架EMM-TTS，通过解耦情感与音色并引入SCL和SEALN模块，实现了更优的自然度、情感迁移性与音色一致性。

摘要：

跨语言情感文本转语音（TTS）旨在用一种语言生成语音，同时捕捉另一种语言说话人的情感，并保持目标声音的音色。这种跨语言情感语音合成过程面临复杂挑战，需要对情感、音色和语言进行灵活控制。然而，情感与音色在语音信号中高度耦合，使得细粒度控制极具挑战。为解决此问题，我们提出EMM-TTS，一种基于扰动自监督学习（SSL）表示的新型两阶段跨语言情感语音合成框架。在第一阶段，模型显式与隐式地编码韵律线索以捕捉情感表现力；第二阶段则从扰动的SSL表示中恢复音色。我们进一步研究了不同说话人扰动策略（共振峰偏移和说话人匿名化）对情感与音色解耦的影响。为增强说话人保持与表现力控制，我们引入了说话人一致性损失（SCL）和说话人-情感自适应层归一化（SEALN）模块。此外，我们发现将显式声学特征（如F0、能量和时长）与预训练潜在特征结合，可提升声音克隆性能。包含主观与客观度量的全面多指标评估表明，EMM-TTS在跨语言场景下实现了更优的自然度、情感可迁移性与音色一致性。

英文摘要：

Cross-lingual emotional text-to-speech (TTS) aims to produce speech in one language that captures the emotion of a speaker from another language while maintaining the target voice's timbre. This process of cross-lingual emotional speech synthesis presents a complex challenge, necessitating flexible control over emotion, timbre, and language. However, emotion and timbre are highly entangled in speech signals, making fine-grained control challenging. To address this issue, we propose EMM-TTS, a novel two-stage cross-lingual emotional speech synthesis framework based on perturbed self-supervised learning (SSL) representations. In the first stage, the model explicitly and implicitly encodes prosodic cues to capture emotional expressiveness, while the second stage restores the timbre from perturbed SSL representations. We further investigate the effect of different speaker perturbation strategies-formant shifting and speaker anonymization-on the disentanglement of emotion and timbre. To strengthen speaker preservation and expressive control, we introduce Speaker Consistency Loss (SCL) and Speaker-Emotion Adaptive Layer Normalization (SEALN) modules. Additionally, we find that incorporating explicit acoustic features (e.g., F0, energy, and duration) alongside pretrained latent features improves voice cloning performance. Comprehensive multi-metric evaluations, including both subjective and objective measures, demonstrate that EMM-TTS achieves superior naturalness, emotion transferability, and timbre consistency across languages.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。