In this paper, we propose a new compositional tool that will generate a musical outline of speech recorded/provided by the user for use as a musical building block in their compositions. The tool allows any user to use their own speech to generate musical material, while still being able to hear the direct connection between their recorded speech and the resulting music. The tool is built on our proposed pipeline. This pipeline begins with speech-based signal processing, after which some simple musical heuristics are applied, and finally these pre-processed signals are passed through Transformer models trained on new musical tasks. We illustrate the effectiveness of our pipeline -- which does not require a paired dataset for training -- through examples of music created by musicians making use of our tool.
A Dhrupad vocal concert comprises a composition section that is interspersed with improvised episodes of increased rhythmic activity involving the interaction between the vocals and the percussion. Tracking the changing rhythmic density, in relation to the underlying metric tempo of the piece, thus facilitates the detection and labeling of the improvised sections in the concert structure. This work concerns the automatic detection of the musically relevant rhythmic densities as they change in time across the bandish (composition) performance. An annotated dataset of Dhrupad bandish concert sec…
The application of Compressive sensing approach to the speech and musical signals is considered in this paper. Compressive sensing (CS) is a new approach to the signal sampling that allows signal reconstruction from a small set of randomly acquired samples. This method is developed for the signals that exhibit the sparsity in a certain domain. Here we have observed two sparsity domains: discrete Fourier and discrete cosine transform domain. Furthermore, two different types of audio signals are analyzed in terms of sparsity and CS performance - musical and speech signals. Comparative analysis o…
National Institute of Informatics, Tokyo, Japan（日本东京国立信息学研究所）；Department of Signal Processing and Acoustics, Aalto University, Finland（芬兰阿尔托大学信号处理与声学系）
Recent neural waveform synthesizers such as WaveNet, WaveGlow, and the neural-source-filter (NSF) model have shown good performance in speech synthesis despite their different methods of waveform generation. The similarity between speech and music audio synthesis techniques suggests interesting avenues to explore in terms of the best way to apply speech synthesizers in the music domain. This work compares three neural synthesizers used for musical instrument sounds generation under three scenarios: training from scratch on music data, zero-shot learning from the speech domain, and fine-tuning-…
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Hand in hand with deep learning advancements, algorithms of music composition increase in performance. However, most of the successful models are designed for specific musical structures. Here, we present BachProp, an algorithmic composer that can generate music scores in any style given sufficient training data. To adapt BachProp to a broad range of musical styles, we propose a novel normalized representation of music and train a deep network to predict the note transition probabilities of a given music corpus. In this paper, new music scores sampled by BachProp are compared with the original…
CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction
National University of Singapore（新加坡国立大学）；University of Science and Technology of China（中国科学技术大学）；University of Cambridge（剑桥大学）；University of Toronto（多伦多大学）
Radboud University（拉德堡德大学）；University of Michigan（密歇根大学）
Generating synthesised singing voice with models trained on speech data has many advantages due to the models' flexibility and controllability. However, since the information about the temporal relationship between segments and beats are lacking in speech training data, the synthesised singing may sound off-beat at times. Therefore, the availability of the information on the temporal relationship between speech segments and music beats is crucial. The current study investigated the segment-beat synchronisation in singing data, with hypotheses formed based on the linguistics theories of P-centr…
Texas A&M University（德州农工大学）；Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）；Concordia University（康考迪亚大学）
A divide and conquer strategy for enhancement of noisy speeches in adverse environments involving lower levels of SNR is presented in this paper, where the total system of speech enhancement is divided into two separate steps. The first step is based on noise compensation on short time magnitude and the second step is based on phase compensation. The magnitude spectrum is compensated based on a modified spectral subtraction method where the cross-terms containing spectra of noise and clean speech are taken into consideration, which are neglected in the traditional spectral subtraction methods.…
Research Center for IT Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）；KKBOX Inc.（KKBOX公司）
Music creation involves not only composing the different parts (e.g., melody, chords) of a musical work but also arranging/selecting the instruments to play the different parts. While the former has received increasing attention, the latter has not been much investigated. This paper presents, to the best of our knowledge, the first deep learning models for rearranging music of arbitrary genres. Specifically, we build encoders and decoders that take a piece of polyphonic musical audio as input and predict as output its musical score. We investigate disentanglement techniques such as adversarial…
Edwin Schlossberg Incorporated（埃德温·施洛斯伯格公司）；EAR Studio（EAR工作室）；Eloise W. Martin Center (ECHO)（埃洛伊丝·W·马丁中心（ECHO））；Interval Research Corporation（英特尔研究公司）
This paper describes a series of projects that explore the possibilities of musical expression through the combination of pre-composed, interlocking, modular components. In particular, this paper presents a modular soundtrack recently composed by the author for "Currents of Creativity," a permanent interactive video wall installation at the Pope John Paul II Cultural Center which is slated to open Easter 2001 in Washington, DC.
There are many applications that aim to create a complete model for an autonomously generated composition; systems are able to generate muzak songs, assist singers in transcribing songs or can imitate long-dead authors. Subjective understanding of creativity or aesthetics differs not only within preferences (popular authors or genres), but also differs on the basis of experienced experience or socio-cultural environment. So, what do we want to achieve with such an adaptation? What is the benefit of the resulting work for the author, who can no longer evaluate this composition? And in what ways…
Many previous audio-visual voice-related works focus on speech, ignoring the singing voice in the growing number of musical video streams on the Internet. For processing diverse musical video data, voice activity detection is a necessary step. This paper attempts to detect the speech and singing voices of target performers in musical video streams using audiovisual information. To integrate information of audio and visual modalities, a multi-branch network is proposed to learn audio and image representations, and the representations are fused by attention based on semantic similarity to shape …
Interdisciplinary Centre for Computer Music Research, University of Plymouth（普利茅斯大学跨学科计算机音乐研究中心）；Oxford Risk Research & Analysis Ltd（牛津风险研究与分析有限公司）；ARM（安谋科技公司）
This paper presents and tests a new approach to composing for ensemble singing performance: reality opera. In the performance of such a composition, emotions of the singers are real and emerge as a consequence of their interactions and reaction and to a dynamic narrative. This paper gives background and motivation for the form, based on three key concepts, incorporating the use of technology. Then proposed techniques for creating reality opera are instantiated in an example, which is performed and a behavioral analysis done of performer reactions, leading to support for the feasibility of the …
Quantum computing is emerging as an alternative computing technology, which is built on the principles of subatomic physics. In spite of continuing progress in developing increasingly more sophisticated hardware and software, access to quantum computing still requires specialist expertise that is largely confined to research laboratories. Moreover, the target applications for these developments remain primarily scientific. This chapter introduces research aimed at improving this scenario. Our research is aimed at extending the range of applications of quantum computing towards the arts and cre…
Music Technology Group, Universitat Pompeu Fabra（庞培法布拉大学音乐技术组）；Joint Research Centre, European Commission（欧盟委员会联合研究中心）
Loops, seamlessly repeatable musical segments, are a cornerstone of modern music production. Contemporary artists often mix and match various sampled or pre-recorded loops based on musical criteria such as rhythm, harmony and timbral texture to create compositions. Taking such criteria into account, we present LoopNet, a feed-forward generative model for creating loops conditioned on intuitive parameters. We leverage Music Information Retrieval (MIR) models as well as a large collection of public loop samples in our study and use the Wave-U-Net architecture to map control parameters to audio. …
Generating a complex work of art such as a musical composition requires exhibiting true creativity that depends on a variety of factors that are related to the hierarchy of musical language. Music generation have been faced with Algorithmic methods and recently, with Deep Learning models that are being used in other fields such as Computer Vision. In this paper we want to put into context the existing relationships between AI-based music composition models and human musical composition and creativity processes. We give an overview of the recent Deep Learning models for music composition and we…
Music is an expressive form of communication often used to convey emotion in scenarios where "words are not enough". Part of this information lies in the musical composition where well-defined language exists. However, a significant amount of information is added during a performance as the musician interprets the composition. The performer injects expressiveness into the written score through variations of different musical properties such as dynamics and tempo. In this paper, we describe a model that can learn to perform sheet music. Our research concludes that the generated performances are…
Stevens Institute of Technology（史蒂文斯理工学院）；University of Arizona（亚利桑那大学）
Most musical programming languages are developed purely for coding virtual instruments or algorithmic compositions. Although there has been some work in the domain of musical query languages for music information retrieval, there has been little attempt to unify the principles of musical programming and query languages with cognitive and natural language processing models that would facilitate the activity of composition by conversation. We present a prototype framework, called MusECI, that merges these domains, permitting score-level algorithmic composition in a text editor while also support…
We present a model for capturing musical features and creating novel sequences of music, called the Convolutional Variational Recurrent Neural Network. To generate sequential data, the model uses an encoder-decoder architecture with latent probabilistic connections to capture the hidden structure of music. Using the sequence-to-sequence model, our generative model can exploit samples from a prior distribution and generate a longer sequence of music. We compare the performance of our proposed model with other types of Neural Networks using the criteria of Information Rate that is implemented by…
This paper presents the Computoser hybrid probability/rule based algorithm for music composition (http://computoser.com) and provides a reference implementation. It addresses the issues of unpleasantness and lack of variation exhibited by many existing approaches by combining the two methods (basing the parameters of the rules on data obtained from preliminary analysis). A sample of 500+ musical pieces was analyzed to derive probabilities for musical characteristics and events (e.g. scale, tempo, intervals). The algorithm was constructed to produce musical pieces using the derived probabilitie…
In the task of generating music, the art factor plays a big role and is a great challenge for AI. Previous work involving adversarial training to produce new music pieces and modeling the compatibility of variety in music (beats, tempo, musical stems) demonstrated great examples of learning this task. Though this was limited to generating mashups or learning features from tempo and key distributions to produce similar patterns. Compound Word Transformer was able to represent music generation task as a sequence generation challenge involving musical events defined by compound words. These music…
Indian Institute of Technology, Madras（印度理工学院马德拉斯分校）
In the domain of algorithmic music composition, machine learning-driven systems eliminate the need for carefully hand-crafting rules for composition. In particular, the capability of recurrent neural networks to learn complex temporal patterns lends itself well to the musical domain. Promising results have been observed across a number of recent attempts at music composition using deep RNNs. These approaches generally aim at first training neural networks to reproduce subsequences drawn from existing songs. Subsequently, they are used to compose music either at the audio sample-level or at the…
Decoding Musical Evolution Through Network Science
Department of Computer Science, Sapienza University of Rome（罗马大学计算机科学系）；Department of Computer, Control and Management Engineering, Sapienza University of Rome（罗马大学计算机、控制与管理工程系）；University of Padova, Department of Mathematics（帕多瓦大学数学系）
Discovering "Words" in Music: Unsupervised Learning of Compositional Sparse Code for Symbolic Music
Central Conservatory of Music（中央音乐学院）；Department of Industrial Engineering, Ajou University（Ajou大学工业工程系）；Alibaba Group（阿里巴巴集团）；Bigai；Peking University（北京大学）
Electronic music artists and sound designers have unique workflow practices that necessitate specialized approaches for developing music information retrieval and creativity support tools. Furthermore, electronic music instruments, such as modular synthesizers, have near-infinite possibilities for sound creation and can be combined to create unique and complex audio paths. The process of discovering interesting sounds is often serendipitous and impossible to replicate. For this reason, many musicians in electronic genres record audio output at all times while they work in the studio. Subsequen…
A musical performance renders an acoustic realization of a musical score or other representation of a composition. Different performances of the same composition may vary in terms of performance parameters such as timing or dynamics, and these variations may have a major impact on how a listener perceives the music. The analysis of music performance has traditionally been a peripheral topic for the MIR research community, where often a single audio recording is used as representative of a musical work. This paper surveys the field of Music Performance Analysis (MPA) from several perspectives i…
Universidad Tecnológica de la Mixteca（米斯特卡科技大学）；University of Minnesota（明尼苏达大学）
In this paper, first musical compositions are presented, which are created using the mathematical counterpoint theory of Guerino Mazzola and his collaborators. These compositions also use the RUBATO(R) software's components for counterpoint constructions. The present work aims at opening new "exotic" directions of contrapuntal composition in non-Fuxian worlds. The authors would like to receive first impressions about these compositions, which are available as scores and audio files.
ATR Media Information Science Labs（ATR媒体信息科学实验室）；RWTH Aachen University（亚琛工业大学）；McGill University（麦吉尔大学）
The role of the face and mouth in speech production as well asnon-verbal communication suggests the use of facial action tocontrol musical sound. Here we document work on theMouthesizer, a system which uses a headworn miniaturecamera and computer vision algorithm to extract shapeparameters from the mouth opening and output these as MIDIcontrol changes. We report our experience with variousgesture-to-sound mappings and musical applications, anddescribe a live performance which used the Mouthesizerinterface.
Vocoders are models capable of transforming a low-dimensional spectral representation of an audio signal, typically the mel spectrogram, to a waveform. Modern speech generation pipelines use a vocoder as their final component. Recent vocoder models developed for speech achieve a high degree of realism, such that it is natural to wonder how they would perform on music signals. Compared to speech, the heterogeneity and structure of the musical sound texture offers new challenges. In this work we focus on one specific artifact that some vocoder models designed for speech tend to exhibit when appl…
Instituto de Investigaciones en Matemáticas Aplicadas y en Sistemas, Universidad Nacional Autónoma de México（墨西哥国立自治大学应用数学与系统研究所）
Topological data analysis has been recently applied to investigate stylistic signatures and trends in musical compositions. A useful tool in this area is Persistent Homology. In this paper, we develop a novel method to represent a weighted directed graph as a finite metric space and then use persistent homology to extract useful features. We apply this method to weighted directed graphs obtained from pitch transitions information of a given musical fragment and use these techniques to the study of stylistic trends. In particular, we are interested in using these tools to make quantitative styl…
This paper introduces a novel recurrent model for music composition that is tailored to the structure of polyphonic music. We propose an efficient new conditional probabilistic factorization of musical scores, viewing a score as a collection of concurrent, coupled sequences: i.e. voices. To model the conditional distributions, we borrow ideas from both convolutional and recurrent neural models; we argue that these ideas are natural for capturing music's pitch invariances, temporal structure, and polyphony. We train models for single-voice and multi-voice composition on 2,300 scores from the Ke…
The term jitter and shimmer has long been used in the domain of speech and acoustic signal analysis as a parameter for speaker identification and other prosodic features. In this study, we look forward to use the same parameters in neural domain to identify and categorize emotional cues in different musical clips. For this, we chose two ragas of Hindustani music which are conventionally known to portray contrast emotions and EEG study was conducted on 5 participants who were made to listen to 3 min clip of these two ragas with sufficient resting period in between. The neural jitter and shimmer…
This paper presents a generative AI model for automated music composition with LSTM networks that takes a novel approach at encoding musical information which is based on movement in music rather than absolute pitch. Melodies are encoded as a series of intervals rather than a series of pitches, and chords are encoded as the set of intervals that each chord note makes with the melody at each timestep. Experimental results show promise as they sound musical and tonal. There are also weaknesses to this method, mainly excessive modulations in the compositions, but that is expected from the nature …
A method is proposed which enables one to produce musical compositions by using transposition in place of harmonic progression. A transposition scale is introduced to provide a set of intervals commensurate with the musical scale, such as chromatic or just intonation scales. A sequence of intervals selected from the transposition scale is used to shift instrument frequency at predefined times during the composition which serves as a harmonic sequence of a composition. A transposition sequence constructed in such a way can be extended to a hierarchy of sequences. The fundamental sound frequency…
JOHANNES KEPLER UNIVERSITY LINZ（约翰内斯·开普勒大学林茨分校）；Institute of Computational Perception（计算感知研究所）
In recent years, artificial neural networks (ANNs) have become a universal tool for tackling real-world problems. ANNs have also shown great success in music-related tasks including music summarization and classification, similarity estimation, computer-aided or autonomous composition, and automatic music analysis. As structure is a fundamental characteristic of Western music, it plays a role in all these tasks. Some structural aspects are particularly challenging to learn with current ANN architectures. This is especially true for mid- and high-level self-similarity, tonal and rhythmic relati…
Distant speech recognition is being revolutionized by deep learning, that has contributed to significantly outperform previous HMM-GMM systems. A key aspect behind the rapid rise and success of DNNs is their ability to better manage large time contexts. With this regard, asymmetric context windows that embed more past than future frames have been recently used with feed-forward neural networks. This context configuration turns out to be useful not only to address low-latency speech recognition, but also to boost the recognition performance under reverberant conditions. This paper investigates …
Mode-conditioned music learning and composition: a spiking neural network inspired by neuroscience and psychology
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Chinese Academy of Sciences（中国科学院脑认知与类脑智能技术重点实验室）
Department of Information and Computing Sciences, Utrecht University（乌得勒支大学信息与计算科学系）；Utrecht University（乌得勒支大学）
Pattern discovery algorithms in the music domain aim to find meaningful components in musical compositions. Over the years, although many algorithms have been developed for pattern discovery in music data, it remains a challenging task. To gain more insight into the efficacy of these algorithms, we introduce three computational methods for examining their output: Pattern Polling, to combine the patterns; Comparative Classification, to differentiate the patterns; Synthetic Data, to inject predetermined patterns. In combining and differentiating the patterns extracted by algorithms, we expose ho…
