Drop the beat! 用于伴奏条件化说唱语音生成的Freestyler
Drop the beat! Freestyler for Accompaniment Conditioned Rapping Voice Generation
AI总结:
本文提出首个直接从歌词和伴奏生成说唱人声的系统Freestyler,结合语言模型、条件流匹配和神经声码器实现零样本音色控制,并构建了RapBank数据集,实验证明其生成质量高且节奏对齐强。
摘要:
说唱作为一种重要的声乐表演流派,在语音生成领域仍缺乏充分探索。通用语音合成依赖于精确的音符和时长输入,要求用户具备相关音乐知识,这限制了灵活性。相比之下,说唱通常具有更简单的旋律,其核心在于与伴奏节拍相和谐的强烈节奏感。在本文中,我们提出了Freestyler,这是首个直接从歌词和伴奏输入生成说唱人声的系统。Freestyler利用基于语言模型的token生成,随后使用条件流匹配模型生成频谱图,并利用神经声码器恢复音频。它允许使用3秒的提示来实现零样本音色控制。由于公开可用的说唱数据集稀缺,我们还发布了RapBank,这是一个从互联网上收集的说唱歌曲数据集,并配有精心设计的处理流程。实验结果表明,Freestyler能够生成高质量的说唱语音,增强了自然度,并在风格和节奏上与伴奏节拍保持高度一致。
英文摘要:
Rap, a prominent genre of vocal performance, remains underexplored in vocal generation. General vocal synthesis depends on precise note and duration inputs, requiring users to have related musical knowledge, which limits flexibility. In contrast, rap typically features simpler melodies, with a core focus on a strong rhythmic sense that harmonizes with accompanying beats. In this paper, we propose Freestyler, the first system that generates rapping vocals directly from lyrics and accompaniment inputs. Freestyler utilizes language model-based token generation, followed by a conditional flow matching model to produce spectrograms and a neural vocoder to restore audio. It allows a 3-second prompt to enable zero-shot timbre control. Due to the scarcity of publicly available rap datasets, we also present RapBank, a rap song dataset collected from the internet, alongside a meticulously designed processing pipeline. Experimental results show that Freestyler produces high-quality rapping voice generation with enhanced naturalness and strong alignment with accompanying beats, both stylistically and rhythmically.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。