基于VITS且结合Whisper与多尺度F0建模的歌声转换
VITS-Based Singing Voice Conversion Leveraging Whisper and multi-scale F0 Modeling
AI总结：
该研究提出基于VITS的歌声转换系统，结合Whisper提取瓶颈特征、多尺度F0建模与三阶段训练策略，在2023歌声转换挑战赛的自然度评测中取得优异成绩。
摘要：
本文介绍了T23团队提交至2023年歌声转换挑战赛（Singing Voice Conversion Challenge 2023）的系统。我们遵循“识别-合成”框架，歌声转换模型以VITS为基础，包含四个核心模块：先验编码器、后验编码器、解码器，以及并行转置卷积组（PBTC）模块。我们特别利用功能强大的预训练ASR模型Whisper提取瓶颈特征（BNF），作为先验编码器的输入。在提取BNF之前，我们对源信号进行音高扰动以去除说话人音色，有效避免了源说话人音色向目标端泄露。此外，PBTC模块提取多尺度F0作为先验编码器的辅助输入，从而更好地捕捉歌声的音高变化。我们设计了三阶段训练策略，以在目标说话人数据有限的情况下，让基础模型更好地适配目标说话人。官方挑战赛结果显示，我们的系统在自然度方面表现优异，在任务1和任务2中分别排名第1和第2。进一步的消融实验验证了我们系统设计的有效性。
英文摘要：
This paper introduces the T23 team's system submitted to the Singing Voice Conversion Challenge 2023. Following the recognition-synthesis framework, our singing conversion model is based on VITS, incorporating four key modules: a prior encoder, a posterior encoder, a decoder, and a parallel bank of transposed convolutions (PBTC) module. We particularly leverage Whisper, a powerful pre-trained ASR model, to extract bottleneck features (BNF) as the input of the prior encoder. Before BNF extraction, we perform pitch perturbation to the source signal to remove speaker timbre, which effectively avoids the leakage of the source speaker timbre to the target. Moreover, the PBTC module extracts multi-scale F0 as the auxiliary input to the prior encoder, thereby capturing better pitch variations of singing. We design a three-stage training strategy to better adapt the base model to the target speaker with limited target speaker data. Official challenge results show that our system has superior performance in naturalness, ranking 1st and 2nd respectively in Task 1 and 2. Further ablation justifies the effectiveness of our system design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
