High-Level Control of Drum Track Generation Using Learned Patterns of Rhythmic Interaction
Sony Computer Science Laboratories (CSL)（索尼计算机科学实验室（CSL））
Stefan Lattner, Maarten Grachten
备注：
Paper accepted at the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA 2019), New Paltz, New York, U.S.A., October 20-23; 6 pages, 3 figures, 1 table
Spurred by the potential of deep learning, computational music generation has gained renewed academic interest. A crucial issue in music generation is that of user control, especially in scenarios where the music generation process is conditioned on existing musical material. Here we propose a model for conditional kick drum track generation that takes existing musical material as input, in addition to a low-dimensional code that encodes the desired relation between the existing material and the new material to be generated. These relational codes are learned in an unsupervised manner from a music dataset. We show that codes can be sampled to create a variety of musically plausible kick drum tracks and that the model can be used to transfer kick drum patterns from one song to another. Lastly, we demonstrate that the learned codes are largely invariant to tempo and time-shift.
Drum-to-Vocal Percussion Sound Conversion and Its Evaluation Methodology
Graduate School of Information Science and Technology, University of Tokyo（东京大学信息科学与技术研究生院）；National Institute of Advanced Industrial Science and Technology (AIST)（国家先进工业科学与技术研究院）；Faculty of Science and Technology, Keio University（庆应大学理工学院）
Academia Sinica（中央研究院）；National Cheng Kung University（台湾成功大学）；Taiwan AI Labs（台湾人工智能实验室）
This paper proposes a model that generates a drum track in the audio domain to play along to a user-provided drum-free recording. Specifically, using paired data of drumless tracks and the corresponding human-made drum tracks, we train a Transformer model to improvise the drum part of an unseen drumless recording. We combine two approaches to encode the input audio. First, we train a vector-quantized variational autoencoder (VQ-VAE) to represent the input audio with discrete codes, which can then be readily used in a Transformer. Second, using an audio-domain beat tracking model, we compute be…
SyncTrack: Rhythmic Stability and Synchronization in Multi-Track Music Generation
Department of Mathematics, The Hong Kong University of Science and Technology（香港科学与技术大学数学系）；State Key Lab of CAD&CG, Zhejiang University（浙江大学计算机辅助设计与图形学国家重点实验室）；Academy of Interdisciplinary Studies, The Hong Kong University of Science and Technology（香港科学与技术大学跨学科研究学院）；State Key Laboratory of Nervous System Disorders, The Hong Kong University of Science and Technology（香港科学与技术大学神经系统紊乱国家重点实验室）；The University of Hong Kong（香港大学）
National Institute of Science and Technology for Complex Systems（国家复杂系统科学技术研究所）
Online music databases have increased signicantly as a consequence of the rapid growth of the Internet and digital audio, requiring the development of faster and more efficient tools for music content analysis. Musical genres are widely used to organize music collections. In this paper, the problem of automatic music genre classification is addressed by exploring rhythm-based features obtained from a respective complex network representation. A Markov model is build in order to analyse the temporal sequence of rhythmic notation events. Feature analysis is performed by using two multivariate st…
Hochschule für Kunst, Design und Populäre Musik (hKDM) Freiburg（弗莱堡艺术、设计与流行音乐学院）
This paper addresses a question about music cognition: how do we derive polymetric structures. A preference rule system is presented which is implemented into a drum computer. The preference rule system allows inferring local polymetric structures, like two-over-three and three-over-two. By analyzing the micro-timing of West African percussion music a timing pattern consisting of six pulses was discovered. It integrates binary and ternary rhythmic feels. The presented drum computer integrates the discovered superimposed polymetric swing (timing and velocity) appropriate to the rhythmic sequenc…
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Stanford University（斯坦福大学）
Creating a complex work of art like music necessitates profound creativity. With recent advancements in deep learning and powerful models such as transformers, there has been huge progress in automatic music generation. In an accompaniment generation context, creating a coherent drum pattern with apposite fills and improvisations at proper locations in a song is a challenging task even for an experienced drummer. Drum beats tend to follow a repetitive pattern through stanzas with fills or improvisation at section boundaries. In this work, we tackle the task of drum pattern generation condition…
National Cheng Kung University（国立成功大学）；Academia Sinica（中央研究院）；Yating Music Team（雅婷音乐团队）；Taiwan AI Labs（台湾人工智能实验室）
This paper presents a novel system architecture that integrates blind source separation with joint beat and downbeat tracking in musical audio signals. The source separation module segregates the percussive and non-percussive components of the input signal, over which beat and downbeat tracking are performed separately and then the results are aggregated with a learnable fusion mechanism. This way, the system can adaptively determine how much the tracking result for an input signal should depend on the input's percussive or non-percussive components. Evaluation on four testing sets that featur…
In this chapter a quantum music generation application called QuiKo will be discussed. It combines existing quantum algorithms with data encoding methods from quantum machine learning to build drum and audio sample patterns from a database of audio tracks. QuiKo leverages the physical properties and characteristics of quantum computers to generate what can be referred to as Soft Rules proposed by Alexis Kirke. These rules take advantage of the noise produced by quantum devices to develop flexible rules and grammars for quantum music generation. These properties include qubit decoherence and ph…
Research Center for Information Technology Innovation, Academia Sinica（中央研究院信息技术创新研究中心）；Department of Computer Science, National Tsing Hua University（国立清华大学计算机科学系）
Generating music has a few notable differences from generating images and videos. First, music is an art of time, necessitating a temporal model. Second, music is usually composed of multiple instruments/tracks with their own temporal dynamics, but collectively they unfold over time interdependently. Lastly, musical notes are often grouped into chords, arpeggios or melodies in polyphonic music, and thereby introducing a chronological ordering of notes is not naturally suitable. In this paper, we propose three models for symbolic multi-track music generation under the framework of generative ad…
Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization
Sound and Music Computing Lab, School of Computing, NUS（新加坡国立大学计算机学院声音与音乐计算实验室）；Courant Institute of Mathematical Sciences, New York University（纽约大学应用数学科学学院）；Music X Lab, MBZUAI（MBZUAI音乐X实验室）
BeatFM: Improving Beat Tracking with Pre-trained Music Foundation Model
School of Computer Science, Fudan University, Shanghai, China（复旦大学计算机科学学院）；Naval Medical Center, PLA, China（中国人民解放军海军医疗中心）；Graduate School of Informatics, Kyoto University, Kyoto, Japan（京都大学信息科学研究生院）；Graduate School of Advanced Science and Engineering, Hiroshima University, Hiroshima, Japan（广岛大学先进科学与工程研究生院）；Shanghai Key Laboratory of Intelligent Information Processing, Fudan University, Shanghai, China（上海智能信息处理重点实验室）
Institute of Systematic Musicology, University of Hamburg（汉堡大学系统音乐学研究所）；Hamburg University of Applied Sciences（汉堡应用科学大学）
When musicians perform in an ensemble, synchronizing to a mutual pace is the foundation of their musical interaction. Clock generators, e.g., metronomes, or drum machines, might assist such synchronization, but these means, in general, will also distort this natural, self-organized, inter-human synchronization process. In this work, the synchronization of musicians to an external rhythm is modeled using the Impulse Pattern Formulation (IPF), an analytical modeling approach for synergetic systems motivated by research on musical instruments. Nonlinear coupling of system components is described …
Music is a form of expression that often requires interaction between players. If one wishes to interact in such a musical way with a computer, it is necessary for the machine to be able to interpret the input given by the human to find its musical meaning. In this work, we propose a system capable of detecting basic rhythmic features that can allow an application to synchronize its output with the rhythm given by the user, without having any prior agreement or requirement on the possible input. The system is described in detail and an evaluation is given through simulation using quantitative …
U.I. for Computing Research（计算研究所）；University of Alicante（阿利坎特大学）
We present a framework based on neural networks to extract music scores directly from polyphonic audio in an end-to-end fashion. Most previous Automatic Music Transcription (AMT) methods seek a piano-roll representation of the pitches, that can be further transformed into a score by incorporating tempo estimation, beat tracking, key estimation or rhythm quantization. Unlike these methods, our approach generates music notation directly from the input audio in a single stage. For this, we use a Convolutional Recurrent Neural Network (CRNN) with Connectionist Temporal Classification (CTC) loss fu…
Sony CSL Paris（索尼计算机科学实验室巴黎分部）；IRCAM（法国声学/音乐研究和协作学院）
In this work, we introduce a system for real-time generation of drum sounds. This system is composed of two parts: a generative model for drum sounds together with a Max4Live plugin providing intuitive controls on the generative process. The generative model consists of a Conditional Wasserstein autoencoder (CWAE), which learns to generate Mel-scaled magnitude spectrograms of short percussion samples, coupled with a Multi-Head Convolutional Neural Network (MCNN) which estimates the corresponding audio signal from the magnitude spectrogram. The design of this model makes it lightweight, so that…
Singapore University of Technology and Design（新加坡科技设计大学）；Institute for Language and Speech Processing, R.C. “Athena”（“雅典娜”研究中心语言与语音处理研究所）
The field of automatic music composition has seen great progress in recent years, specifically with the invention of transformer-based architectures. When using any deep learning model which considers music as a sequence of events with multiple complex dependencies, the selection of a proper data representation is crucial. In this paper, we tackle the task of conditional drums generation using a novel data encoding scheme inspired by the Compound Word representation, a tokenization process of sequential data. Therefore, we present a sequence-to-sequence architecture where a Bidirectional Long …
Creating a good drum track to imitate a skilled performer in digital audio workstations (DAWs) can be a time-consuming process, especially for those unfamiliar with drums. In this work, we introduce PocketVAE, a groove generation system that applies grooves to users' rudimentary MIDI tracks, i.e, templates. Grooves can be either transferred from a reference track, generated randomly or with conditions, such as genres. Our system, consisting of different modules for each groove component, takes a two-step approach that is analogous to a music creation process. First, the note module updates the…
Roli（Roli 公司）；Queen Mary University of London（伦敦玛丽女王大学）
The imitation of percussive sounds via the human voice is a natural and effective tool for communicating rhythmic ideas on the fly. Thus, the automatic retrieval of drum sounds using vocal percussion can help artists prototype drum patterns in a comfortable and quick way, smoothing the creative workflow as a result. Here we explore different strategies to perform this type of query, making use of both traditional machine learning algorithms and recent deep learning techniques. The main hyperparameters from the models involved are carefully selected by feeding performance metrics to a grid sear…
Content-based Controls For Music Large Language Modeling
New York University Shanghai（上海纽约大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Queen Mary University of London（伦敦玛丽女王大学）
Generative artificial intelligence models can be a valuable aid to music composition and live performance, both to aid the professional musician and to help democratize the music creation process for hobbyists. Here we present a novel method that, given an English word or phrase, generates 2 compasses of a 4-piece drum pattern that embodies the "mood" of the given language cue, or that could be used for an audiovisual scene described by the language cue. We envision this tool as composition aid for electronic music and audiovisual soundtrack production, or an improvisation tool for live perfor…
Eindhoven University of Technology（埃因霍温理工大学）；University of Florence（佛罗伦萨大学）；University of Basel（巴塞尔大学）；Conservatorio G.F. Ghedini di Cuneo（库尼奥G.F.盖迪尼音乐学院）；Musica Combinatoria（组合音乐机构）
We describe a system based on deep learning that generates drum patterns in the electronic dance music domain. Experimental results reveal that generated patterns can be employed to produce musically sound and creative transitions between different genres, and that the process of generation is of interest to practitioners in the field.
Inferring music time structures has a broad range of applications in music production, processing and analysis. Scholars have proposed various methods to analyze different aspects of time structures, such as beat, downbeat, tempo and meter. Many state-of-the-art (SOFA) methods, however, are computationally expensive. This makes them inapplicable in real-world industrial settings where the scale of the music collections can be millions. This paper proposes a new state space and a semi-Markov model for music time structure analysis. The proposed approach turns the commonly used 2D state spaces i…
Zhejiang University（浙江大学）；Microsoft Research Asia（微软亚洲研究院）
In pop music, accompaniments are usually played by multiple instruments (tracks) such as drum, bass, string and guitar, and can make a song more expressive and contagious by arranging together with its melody. Previous works usually generate multiple tracks separately and the music notes from different tracks not explicitly depend on each other, which hurts the harmony modeling. To improve harmony, in this paper, we propose a novel MUlti-track MIDI representation (MuMIDI), which enables simultaneous multi-track generation in a single sequence and explicitly models the dependency of the notes f…
ByteDance（字节跳动）；Queen Mary University of London（伦敦玛丽女王大学）
A music mashup combines audio elements from two or more songs to create a new work. To reduce the time and effort required to make them, researchers have developed algorithms that predict the compatibility of audio elements. Prior work has focused on mixing unaltered excerpts, but advances in source separation enable the creation of mashups from isolated stems (e.g., vocals, drums, bass, etc.). In this work, we take advantage of separated stems not just for creating mashups, but for training a model that predicts the mutual compatibility of groups of excerpts, using self-supervised and semi-su…
