Music generation is always interesting in a sense that there is no formalized recipe. In this work, we propose a novel dual-track architecture for generating classical piano music, which is able to model the inter-dependency of left-hand and right-hand piano music. Particularly, we experimented with a lot of different models of neural network as well as different representations of music, and the results show that our proposed model outperforms all other tested methods. Besides, we deployed some special policies for model training and generation, which contributed to the model performance remarkably. Finally, under two evaluation methods, we compared our models with the MuseGAN project and true music.
We propose the Multi-Track Music Machine (MMM), a generative system based on the Transformer architecture that is capable of generating multi-track music. In contrast to previous work, which represents musical material as a single time-ordered sequence, where the musical events corresponding to different tracks are interleaved, we create a time-ordered sequence of musical events for each track and concatenate several tracks into a single sequence. This takes advantage of the Transformer's attention-mechanism, which can adeptly handle long-term dependencies. We explore how various representatio…
Research Center for Information Technology Innovation, Academia Sinica（中央研究院信息技术创新研究中心）；Department of Computer Science, National Tsing Hua University（国立清华大学计算机科学系）
Generating music has a few notable differences from generating images and videos. First, music is an art of time, necessitating a temporal model. Second, music is usually composed of multiple instruments/tracks with their own temporal dynamics, but collectively they unfold over time interdependently. Lastly, musical notes are often grouped into chords, arpeggios or melodies in polyphonic music, and thereby introducing a chronological ordering of notes is not naturally suitable. In this paper, we propose three models for symbolic multi-track music generation under the framework of generative ad…
University of Sussex（萨塞克斯大学）；Singapore University of Technology and Design（新加坡科技设计大学）
We present a Python library, called Midi Miner, that can calculate tonal tension and classify different tracks. MIDI (Music Instrument Digital Interface) is a hardware and software standard for communicating musical events between digital music devices. It is often used for tasks such as music representation, communication between devices, and even music generation [5]. Tension is an essential element of the music listening experience, which can come from a number of musical features including timbre, loudness and harmony [3]. Midi Miner provides a Python implementation for the tonal tension m…
University of Science and Technology of China（中国科学技术大学）
Currently, almost all the multi-track music generation models use the Convolutional Neural Network (CNN) to build the generative model, while the Recurrent Neural Network (RNN) based models can not be applied in this task. In view of the above problem, this paper proposes a RNN-based Hierarchical Multi-modal Fusion Generation Variational Autoencoder (VAE) network, MIDI-Sandwich2, for multi-track symbolic music generation. Inspired by VQ-VAE2, MIDI-Sandwich2 expands the dimension of the original hierarchical model by using multiple independent Binary Variational Autoencoder (BVAE) models withou…
BeatFM: Improving Beat Tracking with Pre-trained Music Foundation Model
School of Computer Science, Fudan University, Shanghai, China（复旦大学计算机科学学院）；Naval Medical Center, PLA, China（中国人民解放军海军医疗中心）；Graduate School of Informatics, Kyoto University, Kyoto, Japan（京都大学信息科学研究生院）；Graduate School of Advanced Science and Engineering, Hiroshima University, Hiroshima, Japan（广岛大学先进科学与工程研究生院）；Shanghai Key Laboratory of Intelligent Information Processing, Fudan University, Shanghai, China（上海智能信息处理重点实验室）
SyncTrack: Rhythmic Stability and Synchronization in Multi-Track Music Generation
Department of Mathematics, The Hong Kong University of Science and Technology（香港科学与技术大学数学系）；State Key Lab of CAD&CG, Zhejiang University（浙江大学计算机辅助设计与图形学国家重点实验室）；Academy of Interdisciplinary Studies, The Hong Kong University of Science and Technology（香港科学与技术大学跨学科研究学院）；State Key Laboratory of Nervous System Disorders, The Hong Kong University of Science and Technology（香港科学与技术大学神经系统紊乱国家重点实验室）；The University of Hong Kong（香港大学）
Recent advancements in generative models have shown remarkable progress in music generation. However, most existing methods focus on generating monophonic or homophonic music, while the generation of polyphonic and multi-track music with rich attributes is still a challenging task. In this paper, we propose a novel approach for multi-track, multi-attribute symphonic music generation using the diffusion model. Specifically, we generate piano-roll representations with a diffusion model and map them to MIDI format for output. To capture rich attribute information, we introduce a color coding sche…
Sony Computer Science Laboratories (CSL)（索尼计算机科学实验室（CSL））
Spurred by the potential of deep learning, computational music generation has gained renewed academic interest. A crucial issue in music generation is that of user control, especially in scenarios where the music generation process is conditioned on existing musical material. Here we propose a model for conditional kick drum track generation that takes existing musical material as input, in addition to a low-dimensional code that encodes the desired relation between the existing material and the new material to be generated. These relational codes are learned in an unsupervised manner from a m…
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Tencent Music Entertainment（腾讯音乐娱乐）
Personalized recommendation on new track releases has always been a challenging problem in the music industry. To combat this problem, we first explore user listening history and demographics to construct a user embedding representing the user's music preference. With the user embedding and audio data from user's liked and disliked tracks, an audio embedding can be obtained for each track using metric learning with Siamese networks. For a new track, we can decide the best group of users to recommend by computing the similarity between the track's audio embedding and different user embeddings, …
Multi-view MidiVAE: Fusing Track- and Bar-view Representations for Long Multi-track Symbolic Music Generation
Tsinghua University（清华大学）；Huawei Technologies Co., Ltd.（华为技术有限公司）；Skywork AI PTE. LTD.（昆仑万维人工智能（Skywork AI））；The Chinese University of Hong Kong（香港中文大学）
University of Science and Technology of China（中国科学技术大学）
Most existing neural network models for music generation explore how to generate music bars, then directly splice the music bars into a song. However, these methods do not explore the relationship between the bars, and the connected song as a whole has no musical form structure and sense of musical direction. To address this issue, we propose a Multi-model Multi-task Hierarchical Conditional VAE-GAN (Variational Autoencoder-Generative adversarial networks) networks, named MIDI-Sandwich, which combines musical knowledge, such as musical form, tonic, and melodic motion. The MIDI-Sandwich has two…
Federal University of Technology – Paraná（巴拉那联邦理工大学）；University of Campinas（坎皮纳斯大学）
Music Genre Classification is the problem of associating genre-related labels to digitized music tracks. It has applications in the organization of commercial and personal music collections. Often, music tracks are described as a set of timbre-inspired sound textures. In shallow-learning systems, the total number of sound textures per track is usually too high, and texture downsampling is necessary to make training tractable. Although previous work has solved this by linear downsampling, no extensive work has been done to evaluate how texture selection benefits genre classification in the cont…
Korea Advanced Institute of Science and Technology（韩国科学技术院）；Stanford University（斯坦福大学）；Seoul National University（首尔大学）
Feature learning and deep learning have drawn great attention in recent years as a way of transforming input data into more effective representations using learning algorithms. Such interest has grown in the area of music information retrieval (MIR) as well, particularly in music audio classification tasks such as auto-tagging. In this paper, we present a two-stage learning model to effectively predict multiple labels from music audio. The first stage learns to project local spectral patterns of an audio track onto a high-dimensional sparse space in an unsupervised manner and summarizes the au…
Previous approaches in singer identification have used one of monophonic vocal tracks or mixed tracks containing multiple instruments, leaving a semantic gap between these two domains of audio. In this paper, we present a system to learn a joint embedding space of monophonic and mixed tracks for singing voice. We use a metric learning method, which ensures that tracks from both domains of the same singer are mapped closer to each other than those of different singers. We train the system on a large synthetic dataset generated by music mashup to reflect real-world music recordings. Our approach…
Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization
Sound and Music Computing Lab, School of Computing, NUS（新加坡国立大学计算机学院声音与音乐计算实验室）；Courant Institute of Mathematical Sciences, New York University（纽约大学应用数学科学学院）；Music X Lab, MBZUAI（MBZUAI音乐X实验室）
Q&A: Query-Based Representation Learning for Multi-Track Symbolic Music re-Arrangement
Institute of Data Science, NUS（新加坡国立大学数据科学研究所）；Integrative Sciences and Engineering Programme, NUS Graduate School（新加坡国立大学研究生院综合科学与工程项目）；Music X Lab, NYU Shanghai（上海纽约大学Music X实验室）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；School of Computing, NUS（新加坡国立大学计算学院）
Virginia Tech（弗吉尼亚理工学院）；Georgia Institute of Technology（佐治亚理工学院）
FusionGAN is a novel genre fusion framework for music generation that integrates the strengths of generative adversarial networks and dual learning. In particular, the proposed method offers a dual learning extension that can effectively integrate the styles of the given domains. To efficiently quantify the difference among diverse domains and avoid the vanishing gradient issue, FusionGAN provides a Wasserstein based metric to approximate the distance between the target domain and the existing domains. Adopting the Wasserstein distance, a new domain is created by combining the patterns of the …
ViTex: Visual Texture Control for Multi-Track Symbolic Music Generation via Discrete Diffusion Models
School of EECS, Peking University（电子工程系，北京大学）；Music X Lab, MBZUAI（音乐X实验室，MBZUAI）；Courant Institute of Mathematical Sciences, New York University（数学科学学院，纽约大学）
As an important format of multimedia, music has filled almost everyone's life. Automatic analyzing music is a significant step to satisfy people's need for music retrieval and music recommendation in an effortless way. Thereinto, downbeat tracking has been a fundamental and continuous problem in Music Information Retrieval (MIR) area. Despite significant research efforts, downbeat tracking still remains a challenge. Previous researches either focus on feature engineering (extracting certain features by signal processing, which are semi-automatic solutions); or have some limitations: they can o…
MusicMamba: A Dual-Feature Modeling Approach for Generating Chinese Traditional Music with Modal Precision
School of Computer Science and Artificial Intelligence, Wuhan University of Technology（武汉理工大学计算机科学与人工智能学院）；School of Computer Science, Hubei University of Technology（湖北工业大学计算机学院）
This paper focuses on the modeling of musical melodies as networks. Notes of a melody can be treated as nodes of a network. Connections are created whenever notes are played in sequence. We analyze some main tracks coming from different music genres, with melodies played using different musical instruments. We find out that the considered networks are, in general, scale free networks and exhibit the small world property. We measure the main metrics and assess whether these networks can be considered as formed by sub-communities. Outcomes confirm that peculiar features of the tracks can be extr…
We introduce MIDI-VAE, a neural network model based on Variational Autoencoders that is capable of handling polyphonic music with multiple instrument tracks, as well as modeling the dynamics of music by incorporating note durations and velocities. We show that MIDI-VAE can perform style transfer on symbolic music by automatically changing pitches, dynamics and instruments of a music piece from, e.g., a Classical to a Jazz style. We evaluate the efficacy of the style transfer by training separate style validation classifiers. Our model can also interpolate between short pieces of music, produce…
Real music signals are highly variable, yet they have strong statistical structure. Prior information about the underlying physical mechanisms by which sounds are generated and rules by which complex sound structure is constructed (notes, chords, a complete musical score), can be naturally unified using Bayesian modelling techniques. Typically algorithms for Automatic Music Transcription independently carry out individual tasks such as multiple-F0 detection and beat tracking. The challenge remains to perform joint estimation of all parameters. We present a Bayesian approach for modelling music…
Artificial intelligence (AI) has been widely applied to music generation topics such as continuation, melody/harmony generation, genre transfer and music infilling application. Although with the burst interest to apply AI to music, there are still few interfaces for the musicians to take advantage of the latest progress of the AI technology. This makes those tools less valuable in practice and harder to find its advantage/drawbacks without utilizing them in the real scenario. This work builds a max patch for interactive music infilling application with different levels of control, including tr…
Local Periodicity-Based Beat Tracking for Expressive Classical Piano Music
National Cheng Kung University（国立成功大学）；Academia Sinica（中央研究院）；International Audio Laboratories Erlangen（埃尔朗根国际音频实验室）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Fraunhofer Institute for Integrated Circuits IIS（弗劳恩霍夫集成电路研究所）；University of Coimbra（科英布拉大学）；Taiwan AI Labs（台湾人工智能实验室）
Musical pieces can be modeled as complex networks. This fosters innovative ways to categorize music, paving the way towards novel applications in multimedia domains, such as music didactics, multimedia entertainment and digital music generation. Clustering these networks through their main metrics allows grouping similar musical tracks. To show the viability of the approach, we provide results on a dataset of guitar solos.
Explicit Tonal Tension Conditioning via Dual-Level Beam Search for Symbolic Music Generation
Artificial Intelligence Lab, Otto-von-Guericke-University, Magdeburg, Germany（奥托·冯·格里克大学人工智能实验室）；University of Porto, Faculty of Engineering and INESC TEC（波尔图大学工程学院）
Central Conservatory of Music（中央音乐学院）；University of Oxford（牛津大学）；Tsinghua University（清华大学）
In this work, we propose a permutation invariant language model, SymphonyNet, as a solution for symbolic symphony music generation. We propose a novel Multi-track Multi-instrument Repeatable (MMR) representation for symphonic music and model the music sequence using a Transformer-based auto-regressive language model with specific 3-D positional embedding. To overcome length overflow when modeling extra-long symphony tokens, we also propose a modified Byte Pair Encoding algorithm (Music BPE) for music tokens and introduce a novel linear transformer decoder architecture as a backbone. Meanwhile,…
In this paper, we present a machine-learning approach to pitch correction for voice in a karaoke setting, where the vocals and accompaniment are on separate tracks and time-aligned. The network takes as input the time-frequency representation of the two tracks and predicts the amount of pitch-shifting in cents required to make the voice sound in-tune with the accompaniment. It is trained on examples of semi-professional singing. The proposed approach differs from existing real-time pitch correction methods by replacing pitch tracking and mapping to a discrete set of notes---for example, the tw…
In the task of generating music, the art factor plays a big role and is a great challenge for AI. Previous work involving adversarial training to produce new music pieces and modeling the compatibility of variety in music (beats, tempo, musical stems) demonstrated great examples of learning this task. Though this was limited to generating mashups or learning features from tempo and key distributions to produce similar patterns. Compound Word Transformer was able to represent music generation task as a sequence generation challenge involving musical events defined by compound words. These music…
MuPT: A Generative Symbolic Music Pretrained Transformer
M-A-P；HKUST（香港科技大学）；University of Manchester（曼彻斯特大学）；Shenzhen Institute of Advanced Technology, CAS（中国科学院深圳先进技术研究院）；QMUL（伦敦玛丽女王大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；MILA（米拉人工智能研究所）；Institute of Automation, CAS（中国科学院自动化研究所）；Central Conservatory of Music（中央音乐学院）；PKU（北京大学）；SJTU（上海交通大学）；MSRA（微软研究院）；University of Waterloo（滑铁卢大学）；Vector Institue（向量研究所）
