具有帧级韵律特征的高度可控的基于扩散的任意到任意语音转换模型
Highly Controllable Diffusion-based Any-to-Any Voice Conversion Model with Frame-level Prosody Feature
AI总结：
提出一种基于扩散模型的任意到任意语音转换系统，通过引入帧级韵律条件模块和自监督后处理步骤，实现对音高、能量和语速的精细控制，在保持语音质量的同时提升了清晰度。
摘要：
我们提出了一种高度可控的语音操作系统，能够同时执行任意到任意语音转换（VC）和韵律调制。最先进的VC系统可以转移句子级特征，如说话人、情感和说话风格。然而，操控帧级韵律（如音高、能量和语速）仍然具有挑战性。我们提出的模型利用帧级韵律特征来有效转移这些属性。具体而言，音高和能量轨迹被整合在韵律条件模块中，然后与说话人和内容嵌入一起输入到基于扩散的解码器中，生成转换后的语音mel频谱图。为了调整语速，我们的系统包含一个基于自监督模型的后处理步骤，从而提高了可控性。与SOTA方法相比，所提出的模型表现出可比的语音质量和提升的清晰度。它能够覆盖不同范围的基础频率（F0）、能量和速度调制，同时保持转换后的语音质量。
英文摘要：
We propose a highly controllable voice manipulation system that can perform any-to-any voice conversion (VC) and prosody modulation simultaneously. State-of-the-art VC systems can transfer sentence-level characteristics such as speaker, emotion, and speaking style. However, manipulating the frame-level prosody, such as pitch, energy and speaking rate, still remains challenging. Our proposed model utilizes a frame-level prosody feature to effectively transfer such properties. Specifically, pitch and energy trajectories are integrated in a prosody conditioning module and then fed alongside speaker and contents embeddings to a diffusion-based decoder generating a converted speech mel-spectrogram. To adjust the speaking rate, our system includes a self-supervised model based post-processing step which allows improved controllability. The proposed model showed comparable speech quality and improved intelligibility compared to a SOTA approach. It can cover a varying range of fundamental frequency (F0), energy and speed modulation while maintaining converted speech quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
