With the goal of building a system capable of controllable symbolic music loop generation and editing, this paper explores a generalisation of Masked Language Modelling we call Superposed Language Modelling. Rather than input tokens being known or unknown, a Superposed Language Model takes priors over the sequence as input, enabling us to apply various constraints to the generation at inference time. After detailing our approach, we demonstrate our model across various editing tasks in the domain of multi-instrument MIDI loops. We end by highlighting some limitations of the approach and avenues for future work. We provides examples from the SLM across multiple generation and editing tasks at https://erl-j.github.io/slm-mml-demo/.
While deep generative models have empowered music generation, it remains a challenging and under-explored problem to edit an existing musical piece at fine granularity. In this paper, we propose SDMuse, a unified Stochastic Differential Music editing and generation framework, which can not only compose a whole musical piece from scratch, but also modify existing musical pieces in many ways, such as combination, continuation, inpainting, and style transferring. The proposed SDMuse follows a two-stage pipeline to achieve music generation and editing on top of a hybrid representation including pi…
The subjective evaluation of music generation techniques has been mostly done with questionnaire-based listening tests while ignoring the perspectives from music composition, arrangement, and soundtrack editing. In this paper, we propose an editing test to evaluate users' editing experience of music generation models in a systematic way. To do this, we design a new music style transfer model combining the non-chronological inference architecture, autoregressive models and the Transformer, which serves as an improvement from the baseline model on the same style transfer task. Then, we compare t…
BeatEdit: Symbolic Music Generation as Explicit Editing
School of Future Technology South China University of Technology Guangzhou China（未来技术学院 华南理工大学 广州 中国）；South China University of Technology（华南理工大学）；Nanjing University（南京大学）
Closing the Loop: PID Feedback Control for Interpretable Activation Steering in Symbolic Music Generation
Athens University of Economics and Business（雅典经济与商业大学）；Orfium Research（Orfium 研究）；Hellenic Mediterranean University（希腊地中海大学）；Archimedes / Athena Research Center（阿基米德/雅典娜研究中心）
Latent Space Disentanglement via Activation Steering for Interpretable Attribute Control in Symbolic Music Generation
Athens University of Economics；Innovation Lab Orfium Athens, Greece；Department of Music Technology；Acoustics Hellenic Mediterranean University Rethymno, Greece；Institute of Informatics \& Telecommunications National Center for Scientific Research “Demokritos” Athens, Greece；Department of Informatics Athens University of Economics
Despite their impressive offline results, deep learning models for symbolic music generation are not widely used in live performances due to a deficit of musically meaningful control parameters and a lack of structured musical form in their outputs. To address these issues we introduce LooperGP, a method for steering a Transformer-XL model towards generating loopable musical phrases of a specified number of bars and time signature, enabling a tool for live coding performances. We show that by training LooperGP on a dataset of 93,681 musical loops extracted from the DadaGP dataset, we are able …
MusicGen-Stem: Multi-stem music generation and edition through autoregressive modeling
Meta；Sorbonne Univ.（索邦大学）；UMR STMS IRCAM-CNRS；Univ. de Lorraine（洛林大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Loria（洛里亚研究所）；Hebrew Univ. of Jerusalem（耶路撒冷希伯来大学）
University of Alicante（阿利坎特大学）；TU Wien（维也纳工业大学）；Queen Mary University of London（伦敦玛丽女王大学）
The International Workshop on Reading Music Systems (WoRMS) is a workshop that tries to connect researchers who develop systems for reading music, such as in the field of Optical Music Recognition, with other researchers and practitioners that could benefit from such systems, like librarians or musicologists. The relevant topics of interest for the workshop include, but are not limited to: Music reading systems; Optical music recognition; Datasets and performance evaluation; Image processing on music scores; Writer identification; Authoring, editing, storing and presentation systems for music …
MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Massachusetts Institute of Technology（麻省理工学院）
In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments. We first identify two key intermediate representations for a successful video to music generator: body keypoints from videos and MIDI events from audio recordings. We then formulate music generation from videos as a motion-to-MIDI translation problem. We present a Graph$-$Transformer framework that can accurately predict MIDI event sequences in accordance with the body movements. The MIDI event can then be converted to realistic music using an …
Microsoft Research Asia（微软亚洲研究院）；Dalian University of Technology（大连理工大学）；Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）；Nanjing University（南京大学）
Multi-Source Contrastive Learning from Musical Audio
Institute of Language and Speech Proc., Athena Research Center（雅典娜研究中心语言与语音研究所）；Institute of Robotics, Athena Research Center（雅典娜研究中心机器人研究所）；School of ECE, National Technical University of Athens（雅典国家技术大学电气与计算机工程学院）
Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation and Editing via Content-based Controls
New York University Shanghai（纽约大学上海分校）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Queen Mary University of London（伦敦大学玛丽皇后学院）
Many social media users prefer consuming content in the form of videos rather than text. However, in order for content creators to produce videos with a high click-through rate, much editing is needed to match the footage to the music. This posts additional challenges for more amateur video makers. Therefore, we propose a novel attention-based model VMT (Video-Music Transformer) that automatically generates piano scores from video frames. Using music generated from models also prevent potential copyright infringements that often come with using existing music. To the best of our knowledge, the…
Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所（Amii））；University of Alberta（阿尔伯塔大学）；University of California, Santa Cruz（加利福尼亚大学圣克鲁兹分校）
This paper presents a new approach for controlling emotion in symbolic music generation with Monte Carlo Tree Search. We use Monte Carlo Tree Search as a decoding mechanism to steer the probability distribution learned by a language model towards a given emotion. At every step of the decoding process, we use Predictor Upper Confidence for Trees (PUCT) to search for sequences that maximize the average values of emotion and quality as given by an emotion classifier and a discriminator, respectively. We use a language model as PUCT's policy and a combination of the emotion classifier and the disc…
LIP6, Université Pierre et Marie Curie（巴黎第六大学 LIP6 实验室）；Sony CSL, Paris（索尼计算机科学实验室（巴黎））；Sony CSL, Japan（索尼计算机科学实验室（日本））
This paper introduces DeepBach, a graphical model aimed at modeling polyphonic music and specifically hymn-like pieces. We claim that, after being trained on the chorale harmonizations by Johann Sebastian Bach, our model is capable of generating highly convincing chorales in the style of Bach. DeepBach's strength comes from the use of pseudo-Gibbs sampling coupled with an adapted representation of musical data. This is in contrast with many automatic music composition approaches which tend to compose music sequentially. Our model is also steerable in the sense that a user can constrain the gen…
