The ability to automatically generate music that appropriately matches an arbitrary input track is a challenging task. We present a novel controllable system for generating single stems to accompany musical mixes of arbitrary length. At the core of our method are audio autoencoders that efficiently compress audio waveform samples into invertible latent representations, and a conditional latent diffusion model that takes as input the latent encoding of a mix and generates the latent encoding of a corresponding stem. To provide control over the timbre of generated samples, we introduce a technique to ground the latent space to a user-provided reference style during diffusion sampling. For further improving audio quality, we adapt classifier-free guidance to avoid distortions at high guidance strengths when generating an unbounded latent space. We train our model on a dataset of pairs of mixes and matching bass stems. Quantitative experiments demonstrate that, given an input mix, the proposed system can generate basslines with user-specified timbres. Our controllable conditional audio generation framework represents a significant step forward in creating generative AI tools to assist musicians in music production.
MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment
Research Center for Information Technology Innovation, Academia Sinica（中央研究院信息技术创新研究中心）；Department of Computer Science, National Tsing Hua University（国立清华大学计算机科学系）
SAMUeL: Efficient Vocal-Conditioned Music Generation via Soft Alignment Attention and Latent Diffusion
Division of Engineering Science University of Toronto（工程科学系 马尼托瓦大学）；Innovative Cognitive Computing Center King Mongkut's University of Technology Thonburi（创新认知计算中心 王后大学技术吞武里）
LaDA-Band: Language Diffusion Models for Vocal-to-Accompaniment Generation
State Key Lab of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences (CAS)（中国科学院计算技术研究所人工智能安全国家重点实验室）；Lyra Lab, Tencent Music Entertainment（腾讯音乐娱乐集团Lyra实验室）；Pengcheng Laboratory（鹏城实验室）
