MPEcho：一种旋律与音素感知的可控翻唱歌曲生成生成框架
MPEcho: A Melody and Phoneme-Aware Generative Framework for Controllable Cover Song Generation
浏览论文内容
中文总结 AI 辅助
本研究针对现有翻唱歌曲生成模型音素错误率高的问题，提出旋律与音素感知的MPEcho框架，集成音素编码器与长度调节器，结合自研Phonsa转录模型，实现了更精准的可控翻唱歌曲生成。
中文摘要 AI 辅助
翻唱歌曲生成（CSG）需在保留参考歌曲的旋律与语言内容的同时，重新创作其余音乐成分。当前最优模型SongEcho利用基频（$F_0$）序列和有声/无声（V/UV）标签进行条件约束，但V/UV标签隐含的语言信息无法保证歌词准确性，导致音素错误率（PER）较高。受歌唱语音合成（SVS）启发，我们提出MPEcho，将音素编码器和长度调节器（LR）集成到SongEcho框架中，通过提供显式音素级条件约束和精确时间边界，大幅降低PER。为此，我们开发了Phonsa，一种基于Whisper的自动转录模型，可为歌唱语音提供高精度音素级标注，解决了高质量音频-音素对稀缺的问题。实验验证了Phonsa在对齐任务和MPEcho在端到端CSG任务中的有效性，音频样本、代码及权重可通过指定URL获取。
英文摘要
Cover song generation (CSG) should preserve the melodic and linguistic content of a reference song while recreating the remaining musical components. The state-of-the-art model SongEcho utilizes $F_0$ sequences and voiced/unvoiced (V/UV) tags for conditioning; however, implicit linguistic information from V/UV tags cannot guarantee lyric accuracy, leading to a high phoneme error rate (PER). Inspired by singing voice synthesis (SVS), we propose MPEcho, which integrates a phoneme encoder and a length regulator (LR) into the SongEcho framework. By providing explicit phoneme-level conditioning and precise temporal boundaries, MPEcho significantly reduces PER. To enable this, we developed Phonsa, a Whisper-based automatic transcription model that provides high-precision phoneme-level annotations for singing voices, overcoming the scarcity of high-quality audio-phoneme pairs. Experimental results validate the effectiveness of Phonsa for alignment and MPEcho for end-to-end CSG. The audio samples, code and weights can be accessed from https://lonian6.github.io/MPEcho.github.io/.
