无文本和非平行的语音到语音情感风格迁移
Textless and Non-Parallel Speech-to-Speech Emotion Style Transfer
AI总结：
本文提出了一种无文本和非平行的语音到语音情感风格迁移框架，通过分析-合成流程实现情感属性转移，保留说话者身份和语音内容，并在情感迁移效果上优于现有方法。
摘要：
给定一对源语音和参考语音录音，语音到语音（S2S）情感风格迁移涉及生成一个输出语音，使其模仿参考语音的情感特征，同时保留源语音的内容和说话者属性。在本文中，我们提出了一种语音到语音零样本情感风格迁移框架，称为S2S零样本情感风格迁移（S2S-ZEST），该框架能够在将参考语音的情感属性转移到源语音的同时保留说话者身份和语音内容。S2S-ZEST框架由一个分析-合成流程组成，其中分析模块从语音中提取语义标记、说话者表示和情感嵌入。利用这些表示，学习了一个音高轮廓估计器和持续时间预测器。进一步，设计了一个合成模块，根据输入表示和推导出的因素生成语音。分析-合成流程通过使用自动编码目标进行训练，以在推理过程中实现高效的重合成。对于S2S情感风格迁移，从参考语音中提取的情感嵌入以及从源语音中提取的其余表示被用于合成模块以生成风格迁移的语音。在我们的实验中，我们评估了转换语音在内容和说话者保留（相对于源）以及情感风格迁移的有效性（相对于参考）方面。所提出的框架在无文本和非平行设置中展示了优于先前方法的情感风格迁移性能。我们还展示了所提出工作在情感识别任务中的数据增强应用。
英文摘要：
Given a pair of source and reference speech recordings, speech-to-speech (S2S) emotion style transfer involves the generation of an output speech that mimics the emotion characteristics of the reference while preserving the content and speaker attributes of the source. In this paper, we propose a speech-to-speech zero-shot emotion style transfer framework, termed S2S Zero-shot Emotion Style Transfer (S2S-ZEST), that enables the transfer of emotional attributes from the reference to the source while retaining the speaker identity and speech content. The S2S-ZEST framework consists of an analysis-synthesis pipeline in which the analysis module extracts semantic tokens, speaker representations, and emotion embeddings from speech. Using these representations, a pitch contour estimator and a duration predictor are learned. Further, a synthesis module is designed to generate speech based on the input representations and the derived factors. The analysis-synthesis pipeline is trained using an auto-encoding objective to enable efficient resynthesis during inference. For S2S emotion style transfer, the emotion embedding extracted from the reference speech along with the remaining representations from the source speech are used in the synthesis module to generate the style-transferred speech. In our experiments, we evaluate the converted speech on content and speaker preservation (with respect to the source) as well as on the effectiveness of the emotion style transfer (with respect to the reference). The proposed framework demonstrates improved emotion style transfer performance over prior methods in a textless and non-parallel setting. We also illustrate the application of the proposed work for data augmentation in emotion recognition tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
