arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.13763cs.SDeess.AS
收藏

基于扩散模型时变反演的音乐风格迁移

Music Style Transfer with Time-Varying Inversion of Diffusion Models

Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Kuaishou Technology(快手科技)

Sifei Li, Yuxin Zhang, Fan Tang, Chongyang Ma, Weiming dong, Changsheng Xu

AI总结:

本文提出基于扩散模型时变文本反演的音乐风格迁移方法,以极少数据捕捉梅尔频谱特征,并通过偏置减少风格化技术实现稳定迁移,可迁移乐器风格并融入自然声音创作旋律。

摘要:

随着扩散模型的发展,文本引导的图像风格迁移已展现出高质量的可控合成结果。然而,利用文本进行多样化的音乐风格迁移面临重大挑战,主要原因是匹配的音频-文本数据集十分有限。音乐作为一种抽象而复杂的艺术形式,即使在同一流派内部也呈现出变化与微妙之处,因此准确的文本描述十分困难。本文提出了一种音乐风格迁移方法,能够在极少数据条件下有效捕捉音乐属性。我们引入了一个新颖的时变文本反演模块,以精确捕捉不同层次的梅尔频谱图特征。在推理阶段,我们提出了一种偏置减少的风格化技术,以获得稳定的结果。实验结果表明,我们的方法能够迁移特定乐器的风格,并能融入自然声音来创作旋律。样本和源代码可在 https://lsfhuihuiff.github.io/MusicTI/ 获取。

英文摘要:

With the development of diffusion models, text-guided image style transfer has demonstrated high-quality controllable synthesis results. However, the utilization of text for diverse music style transfer poses significant challenges, primarily due to the limited availability of matched audio-text datasets. Music, being an abstract and complex art form, exhibits variations and intricacies even within the same genre, thereby making accurate textual descriptions challenging. This paper presents a music style transfer approach that effectively captures musical attributes using minimal data. We introduce a novel time-varying textual inversion module to precisely capture mel-spectrogram features at different levels. During inference, we propose a bias-reduced stylization technique to obtain stable results. Experimental results demonstrate that our method can transfer the style of specific instruments, as well as incorporate natural sounds to compose melodies. Samples and source code are available at https://lsfhuihuiff.github.io/MusicTI/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑