The current wave of deep learning (the hyper-vitamined return of artificial neural networks) applies not only to traditional statistical machine learning tasks: prediction and classification (e.g., for weather prediction and pattern recognition), but has already conquered other areas, such as translation. A growing area of application is the generation of creative content, notably the case of music, the topic of this paper. The motivation is in using the capacity of modern deep learning techniques to automatically learn musical styles from arbitrary musical corpora and then to generate musical samples from the estimated distribution, with some degree of control over the generation. This paper provides a tutorial on music generation based on deep learning techniques. After a short introduction to the topic illustrated by a recent exemple, the paper analyzes some early works from the late 1980s using artificial neural networks for music generation and how their pioneering contributions have prefigured current techniques. Then, we introduce some conceptual framework to analyze the various concepts and dimensions involved. Various examples of recent systems are introduced and analyzed to illustrate the variety of concerns and of techniques.
Max Planck Institute for Mathematics in the Sciences（马克斯·普朗克数学科学研究所）；Yandex（扬德克斯）
A new architecture of an artificial neural network that helps to generate longer melodic patterns is introduced alongside with methods for post-generation filtering. The proposed approach called variational autoencoder supported by history is based on a recurrent highway gated network combined with a variational autoencoder. Combination of this architecture with filtering heuristics allows generating pseudo-live acoustically pleasing and melodically diverse music.
Indian Institute of Technology Kharagpur（印度理工学院卡拉格普尔分校）
Music genre classification is one of the trending topics in regards to the current Music Information Retrieval (MIR) Research. Since, the dependency of genre is not only limited to the audio profile, we also make use of textual content provided as lyrics of the corresponding song. We implemented a CNN based feature extractor for spectrograms in order to incorporate the acoustic features and a Hierarchical Attention Network based feature extractor for lyrics. We then go on to classify the music track based upon the resulting fused feature vector.
Music holds a significant cultural role in social identity and in the encouragement of socialization. Technology, by the destruction of physical and cultural distance, has lead to many changes in musical themes and the complete loss of forms. Yet, it also allows for the preservation and distribution of music from societies without a history of written sheet music. This paper presents early work on a tool for musicians and ethnomusicologists to transcribe sheet music from monophonic voiced pieces for preservation and distribution. Using FFT, the system detects the pitch frequencies, also other …
The utilization of deep learning techniques in generating various contents (such as image, text, etc.) has become a trend. Especially music, the topic of this paper, has attracted widespread attention of countless researchers.The whole process of producing music can be divided into three stages, corresponding to the three levels of music generation: score generation produces scores, performance generation adds performance characteristics to the scores, and audio generation converts scores with performance characteristics into audio by assigning timbre or generates music in audio format directl…
Music genre classification is one of the sub-disciplines of music information retrieval (MIR) with growing popularity among researchers, mainly due to the already open challenges. Although research has been prolific in terms of number of published works, the topic still suffers from a problem in its foundations: there is no clear and formal definition of what genre is. Music categorizations are vague and unclear, suffering from human subjectivity and lack of agreement. In its first part, this paper offers a survey trying to cover the many different aspects of the matter. Its main goal is give …
Structure is one of the most essential aspects of music, and music structure is commonly indicated through repetition. However, the nature of repetition and structure in music is still not well understood, especially in the context of music generation, and much remains to be explored with Music Information Retrieval (MIR) techniques. Analyses of two popular music datasets (Chinese and American) illustrate important music construction principles: (1) structure exists at multiple hierarchical levels, (2) songs use repetition and limited vocabulary so that individual songs do not follow general s…
Music is one of the Gardner's intelligences in his theory of multiple intelligences. How humans perceive and understand music is still being studied and is crucial to develop artificial intelligence models that imitate such processes. Music generation with Artificial Intelligence is an emerging field that is gaining much attention in the recent years. In this paper, we describe how humans compose music and how new AI systems could imitate such process by comparing past and recent advances in the field with music composition techniques. To understand how AI models and algorithms generate music …
Nanjing University（南京大学）；Microsoft Research Asia（微软亚洲研究院）；Peking University（北京大学）
Symbolic music generation aims to generate music scores automatically. A recent trend is to use Transformer or its variants in music generation, which is, however, suboptimal, because the full attention cannot efficiently model the typically long music sequences (e.g., over 10,000 tokens), and the existing models have shortcomings in generating musical repetition structures. In this paper, we propose Museformer, a Transformer with a novel fine- and coarse-grained attention for music generation. Specifically, with the fine-grained attention, a token of a specific bar directly attends to all the…
ComposerX: Multi-Agent Symbolic Music Composition with LLMs
University of Rochester（罗切斯特大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Hong Kong University of Science and Technology（香港科技大学）；University of Surrey（萨里大学）；The University of Manchester（曼彻斯特大学）；Queen Mary University of London（伦敦玛丽女王大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Tencent Music Entertainment（腾讯音乐娱乐）
Personalized recommendation on new track releases has always been a challenging problem in the music industry. To combat this problem, we first explore user listening history and demographics to construct a user embedding representing the user's music preference. With the user embedding and audio data from user's liked and disliked tracks, an audio embedding can be obtained for each track using metric learning with Siamese networks. For a new track, we can decide the best group of users to recommend by computing the similarity between the track's audio embedding and different user embeddings, …
International Institute of Information Technology, Hyderabad（印度海得拉巴国际信息技术学院）；University of Jyväskylä（于韦斯屈莱大学）
Musical preferences have been considered a mirror of the self. In this age of Big Data, online music streaming services allow us to capture ecologically valid music listening behavior and provide a rich source of information to identify several user-specific aspects. Studies have shown musical engagement to be an indirect representation of internal states including internalized symptomatology and depression. The current study aims at unearthing patterns and trends in the individuals at risk for depression as it manifests in naturally occurring music listening behavior. Mental well-being scores…
The use of deep learning to solve problems in literary arts has been a recent trend that has gained a lot of attention and automated generation of music has been an active area. This project deals with the generation of music using raw audio files in the frequency domain relying on various LSTM architectures. Fully connected and convolutional layers are used along with LSTM's to capture rich features in the frequency domain and increase the quality of music generated. The work is focused on unconstrained music generation and uses no information about musical structure(notes or chords) to aid l…
Queen Mary University of London（伦敦玛丽女王大学）；Imperial College London（帝国理工学院）
In modern societies, cultural change seems ceaseless. The flux of fashion is especially obvious for popular music. While much has been written about the origin and evolution of pop, most claims about its history are anecdotal rather than scientific in nature. To rectify this we investigate the US Billboard Hot 100 between 1960 and 2010. Using Music Information Retrieval (MIR) and text-mining tools we analyse the musical properties of ~17,000 recordings that appeared in the charts and demonstrate quantitative trends in their harmonic and timbral properties. We then use these properties to produ…
The focus of this work is to study how to efficiently tailor Convolutional Neural Networks (CNNs) towards learning timbre representations from log-mel magnitude spectrograms. We first review the trends when designing CNN architectures. Through this literature overview we discuss which are the crucial points to consider for efficiently learning timbre representations using CNNs. From this discussion we propose a design strategy meant to capture the relevant time-frequency contexts for learning timbre, which permits using domain knowledge for designing architectures. In addition, one of our main…
Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey
Univ. Lille（里尔大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Centrale Lille（里尔中央理工学院）；Univ. Bordeaux（波尔多大学）；Bordeaux INP（波尔多综合理工学院）；Singapore University of Technology and Design（新加坡科技设计大学）
In this review, it is explained and compared different software and formalisms used in music interaction: sequencers, computer-assisted improvisation, meta- instruments, score-following, asynchronous dataflow languages, synchronous dataflow languages, process calculi, temporal constraints and interactive scores. Formal approaches have the advantage of providing rigorous semantics of the behavior of the model and proving correctness during execution. The main disadvantage of formal approaches is lack of commercial tools.
The big data about music history contains information about time and users' behavior. Researchers could predict the trend of popular songs accurately by analyzing this data. The traditional trend prediction models can better predict the short trend than the long trend. In this paper, we proposed the improved LSTM Rolling Prediction Algorithm (LSTM-RPA), which combines LSTM historical input with current prediction results as model input for next time prediction. Meanwhile, this algorithm converts the long trend prediction task into multiple short trend prediction tasks. The evaluation results s…
Henry Samueli School of Engineering（亨利·萨缪里工程学院）；University of California-Irvine（加州大学欧文分校）
This work proposes a novel feature selection algorithm to classify Songs into different groups. Classification of musical content is often a non-trivial job and still relatively less explored area. The main idea conveyed in this article is to come up with a new feature selection scheme that does the classification job elegantly and with high accuracy but with simpler but wisely chosen small number of features thus being less prone to over-fitting. This uses a very basic general idea about the structure of the audio signal which is generally in the shape of a trapezium. So, using this general i…
Is it possible use algorithms to find trends in the history of popular music? And is it possible to predict the characteristics of future music genres? In order to answer these questions, we produced a hand-crafted dataset with the intent to put together features about style, psychology, sociology and typology, annotated by music genre and indexed by time and decade. We collected a list of popular genres by decade from Wikipedia and scored music genres based on Wikipedia descriptions. Using statistical and machine learning techniques, we find trends in the musical preferences and use time seri…
Centre de Recerca Matemàtica（数学研究中心）；Dolby Laboratories（杜比实验室）；Departament de Matemàtiques, Facultat de Ciències, Universitat Autònoma de Barcelona（巴塞罗那自治大学科学学院数学系）；Barcelona Graduate School of Mathematics（巴塞罗那数学研究生院）；Complexity Science Hub Vienna（维也纳复杂科学中心）
Music is a fundamental human construct, and harmony provides the building blocks of musical language. Using the Kunstderfuge corpus of classical music, we analyze the historical evolution of the richness of harmonic vocabulary of 76 classical composers, covering almost 6 centuries. Such corpus comprises about 9500 pieces, resulting in more than 5 million tokens of music codewords. The fulfilment of Heaps' law for the relation between the size of the harmonic vocabulary of a composer (in codeword types) and the total length of his works (in codeword tokens), with an exponent around 0.35, allows…
Music is a mysterious language that conveys feeling and thoughts via different tones and timbre. For better understanding of timbre in music, we chose music data of 6 representative instruments, analysed their timbre features and classified them. Instead of the current trend of Neural Network for black-box classification, our project is based on a combination of MFCC and LPC, and augmented with a 6-dimensional feature vector designed by ourselves from observation and attempts. In our white-box model, we observed significant patterns of sound that distinguish different timbres, and discovered s…
Instituto de Investigaciones en Matemáticas Aplicadas y en Sistemas, Universidad Nacional Autónoma de México（墨西哥国立自治大学应用数学与系统研究所）
Topological data analysis has been recently applied to investigate stylistic signatures and trends in musical compositions. A useful tool in this area is Persistent Homology. In this paper, we develop a novel method to represent a weighted directed graph as a finite metric space and then use persistent homology to extract useful features. We apply this method to weighted directed graphs obtained from pitch transitions information of a given musical fragment and use these techniques to the study of stylistic trends. In particular, we are interested in using these tools to make quantitative styl…
Rage Music Classification and Analysis using K-Nearest Neighbour, Random Forest, Support Vector Machine, Convolutional Neural Networks, and Gradient Boosting
IRIT（图卢兹计算机科学研究所）；Université de Toulouse（图卢兹大学）；CNRS（法国国家科学研究中心）
State-of-the-art music recommendation systems are based on collaborative filtering, which predicts a user's interest from his listening habits and similarities with other users' profiles. These approaches are agnostic to the song content, and therefore face the cold-start problem: they cannot recommend novel songs without listening history. To tackle this issue, content-aware recommendation incorporates information about the songs that can be used for recommending new items. Most methods falling in this category exploit either user-annotated tags, acoustic features or deeply-learned features. …
Peking University（北京大学）；New York University Shanghai（上海纽约大学）
Led by the success of neural style transfer on visual arts, there has been a rising trend very recently in the effort of music style transfer. However, "music style" is not yet a well-defined concept from a scientific point of view. The difficulty lies in the intrinsic multi-level and multi-modal character of music representation (which is very different from image representation). As a result, depending on their interpretation of "music style", current studies under the category of "music style transfer", are actually solving completely different problems that belong to a variety of sub-field…
Decoding Musical Evolution Through Network Science
Department of Computer Science, Sapienza University of Rome（罗马大学计算机科学系）；Department of Computer, Control and Management Engineering, Sapienza University of Rome（罗马大学计算机、控制与管理工程系）；University of Padova, Department of Mathematics（帕多瓦大学数学系）
Higher-Order Network Representation of J. S. Bach's Solo Violin Sonatas and Partitas: Topological and Geometrical Explorations
Department of Physics, American University of Beirut, Beirut, Lebanon（物理系，贝鲁特美国大学，贝鲁特，黎巴嫩）；Center for Advanced Mathematical Sciences, American University of Beirut, Beirut, Lebanon（先进数学科学中心，贝鲁特美国大学，贝鲁特，黎巴嫩）
National Taiwan University（台湾大学）；KKBOX Inc.（KKBOX公司）
Songs can be well arranged by professional music curators to form a riveting playlist that creates engaging listening experiences. However, it is time-consuming for curators to timely rearrange these playlists for fitting trends in future. By exploiting the techniques of deep learning and reinforcement learning, in this paper, we consider music playlist generation as a language modeling problem and solve it by the proposed attention language model with policy gradient. We develop a systematic and interactive approach so that the resulting playlists can be tuned flexibly according to user prefe…
