arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.07069cs.SDeess.AS
收藏

Music ControlNet:用于音乐生成的多种时变控制

Music ControlNet: Multiple Time-varying Controls for Music Generation

Carnegie Mellon University(卡内基梅隆大学);Adobe Research(Adobe研究院)

Shih-Lun Wu, Chris Donahue, Shinji Watanabe, Nicholas J. Bryan

AI总结:

Music ControlNet提出基于扩散的时变控制音乐生成方法,支持旋律、动态和节奏控制,相比MusicGen在参数少35倍、数据少11倍下旋律忠实度提升49%。

摘要:

文本到音乐生成模型现在能够生成广泛风格的高质量音乐音频。然而,文本控制主要适用于操纵全局音乐属性,如流派、情绪和速度,而不太适合对时变属性进行精确控制,例如节拍在时间上的位置或音乐动态的变化。我们提出了Music ControlNet,一种基于扩散的音乐生成模型,它提供了对生成音频的多种精确时变控制。为了赋予文本到音乐模型时变控制能力,我们提出了一种类似于图像域ControlNet方法的像素级控制的方法。具体来说,我们从训练音频中提取控制,生成配对数据,并在给定旋律、动态和节奏控制的音频频谱图上微调基于扩散的条件生成模型。虽然图像域的Uni-ControlNet方法已经允许使用任何控制子集进行生成,但我们设计了一种新策略,允许创作者输入在时间上仅部分指定的控制。我们评估了从音频中提取的控制以及我们期望创作者提供的控制,证明在这两种设置下,我们都能生成与控制输入相对应的逼真音乐。虽然存在的可比音乐生成模型很少,但我们与MusicGen(一种接受文本和旋律输入的近期模型)进行了基准比较,结果表明,尽管我们的模型参数少35倍,训练数据少11倍,并且支持两种额外的时变控制形式,但我们的模型生成的音乐对输入旋律的忠实度高出49%。声音示例可在https://MusicControlNet.github.io/web/找到。

英文摘要:

Text-to-music generation models are now capable of generating high-quality music audio in broad styles. However, text control is primarily suitable for the manipulation of global musical attributes like genre, mood, and tempo, and is less suitable for precise control over time-varying attributes such as the positions of beats in time or the changing dynamics of the music. We propose Music ControlNet, a diffusion-based music generation model that offers multiple precise, time-varying controls over generated audio. To imbue text-to-music models with time-varying control, we propose an approach analogous to pixel-wise control of the image-domain ControlNet method. Specifically, we extract controls from training audio yielding paired data, and fine-tune a diffusion-based conditional generative model over audio spectrograms given melody, dynamics, and rhythm controls. While the image-domain Uni-ControlNet method already allows generation with any subset of controls, we devise a new strategy to allow creators to input controls that are only partially specified in time. We evaluate both on controls extracted from audio and controls we expect creators to provide, demonstrating that we can generate realistic music that corresponds to control inputs in both settings. While few comparable music generation models exist, we benchmark against MusicGen, a recent model that accepts text and melody input, and show that our model generates music that is 49% more faithful to input melodies despite having 35x fewer parameters, training on 11x less data, and enabling two additional forms of time-varying control. Sound examples can be found at https://MusicControlNet.github.io/web/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑