Application of Optimization and Simulation to Musical Composition that Emerges Dynamically during Ensemble Singing Performance
Interdisciplinary Centre for Computer Music Research, University of Plymouth（普利茅斯大学跨学科计算机音乐研究中心）；Oxford Risk Research & Analysis Ltd（牛津风险研究与分析有限公司）；ARM（安谋科技公司）
This paper presents and tests a new approach to composing for ensemble singing performance: reality opera. In the performance of such a composition, emotions of the singers are real and emerge as a consequence of their interactions and reaction and to a dynamic narrative. This paper gives background and motivation for the form, based on three key concepts, incorporating the use of technology. Then proposed techniques for creating reality opera are instantiated in an example, which is performed and a behavioral analysis done of performer reactions, leading to support for the feasibility of the reality opera concept.
Music information retrieval is currently an active research area that addresses the extraction of musically important information from audio signals, and the applications of such information. The extracted information can be used for search and retrieval of music in recommendation systems, or to aid musicological studies or even in music learning. Sophisticated signal processing techniques are applied to convert low-level acoustic signal properties to musical attributes which are further embedded in a rule-based or statistical classification framework to link with high-level descriptions such …
Radboud University（拉德堡德大学）；University of Michigan（密歇根大学）
Generating synthesised singing voice with models trained on speech data has many advantages due to the models' flexibility and controllability. However, since the information about the temporal relationship between segments and beats are lacking in speech training data, the synthesised singing may sound off-beat at times. Therefore, the availability of the information on the temporal relationship between speech segments and music beats is crucial. The current study investigated the segment-beat synchronisation in singing data, with hypotheses formed based on the linguistics theories of P-centr…
This paper proposes a novel sequence-to-sequence (seq2seq) model with a musical note position-aware attention mechanism for singing voice synthesis (SVS). A seq2seq modeling approach that can simultaneously perform acoustic and temporal modeling is attractive. However, due to the difficulty of the temporal modeling of singing voices, many recent SVS systems with an encoder-decoder-based model still rely on explicitly on duration information generated by additional modules. Although some studies perform simultaneous modeling using seq2seq models with an attention mechanism, they have insufficie…
Universitat Pompeu Fabra（庞培法布拉大学）；Joint Research Centre, European Commission（欧盟委员会联合研究中心）
We present a deep learning based methodology for extracting the singing voice signal from a musical mixture based on the underlying linguistic content. Our model follows an encoder decoder architecture and takes as input the magnitude component of the spectrogram of a musical mixture with vocals. The encoder part of the model is trained via knowledge distillation using a teacher network to learn a content embedding, which is decoded to generate the corresponding vocoder features. Using this methodology, we are able to extract the unprocessed raw vocal signal from the mixture even for a process…
Universitat Pompeu Fabra（庞培法布拉大学）；Joint Research Centre, European Commission（欧盟委员会联合研究中心）
This paper presents a novel method for extracting the vocal track from a musical mixture. The musical mixture consists of a singing voice and a backing track which may comprise of various instruments. We use a convolutional network with skip and residual connections as well as dilated convolutions to estimate vocoder parameters, given the spectrogram of an input mixture. The estimated parameters are then used to synthesize the vocal track, without any interference from the backing track. We evaluate our system, through objective metrics pertinent to audio quality and interference from backgrou…
NLPR, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所自然语言处理实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
End-to-end singing voice synthesis (SVS) is attractive due to the avoidance of pre-aligned data. However, the auto learned alignment of singing voice with lyrics is difficult to match the duration information in musical score, which will lead to the model instability or even failure to synthesize voice. To learn accurate alignment information automatically, this paper proposes an end-to-end SVS framework, named Singing-Tacotron. The main difference between the proposed framework and Tacotron is that the speech can be controlled significantly by the musical score's duration information. Firstly…
Academia Sinica（中央研究院）；National Taiwan University of Science and Technology（台湾科技大学）
The purpose of this study is to investigate how humans interpret musical scores expressively, and then design machines that sing like humans. We consider six factors that have a strong influence on the expression of human singing. The factors are related to the acoustic, phonetic, and musical features of a real singing signal. Given real singing voices recorded following the MIDI scores and lyrics, our analysis module can extract the expression parameters from the real singing signals semi-automatically. The expression parameters are used to control the singing voice synthesis (SVS) system for…
We present SingSong, a system that generates instrumental music to accompany input vocals, potentially offering musicians and non-musicians alike an intuitive new way to create music featuring their own voice. To accomplish this, we build on recent developments in musical source separation and audio generation. Specifically, we apply a state-of-the-art source separation algorithm to a large corpus of music audio to produce aligned pairs of vocals and instrumental sources. Then, we adapt AudioLM (Borsos et al., 2022) -- a state-of-the-art approach for unconditional audio generation -- to be sui…
National Tsing Hua University（国立清华大学）；Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）；Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所）
Singing voice synthesis (SVS) is the computer production of a human-like singing voice from given musical scores. To accomplish end-to-end SVS effectively and efficiently, this work adopts the acoustic model-neural vocoder architecture established for high-quality speech and singing voice synthesis. Specifically, this work aims to pursue a higher level of expressiveness in synthesized voices by combining the diffusion denoising probabilistic model (DDPM) and \emph{Wasserstein} generative adversarial network (WGAN) to construct the backbone of the acoustic model. On top of the proposed acoustic…
DNN-based ensemble singing voice synthesis with interactions between singers
The University of Tokyo（东京大学）；Keio University（庆应义塾大学）；National Institute of Advanced Industrial Science and Technology (AIST)（国立先进工业科学技术研究所（AIST））；Meiji University（明治大学）
Music Technology Group, Universitat Pompeu Fabra（庞培法布拉大学音乐技术集团）；European Commission, Joint Research Centre（欧盟委员会联合研究中心）
Choral singing is a widely practiced form of ensemble singing wherein a group of people sing simultaneously in polyphonic harmony. The most commonly practiced setting for choir ensembles consists of four parts; Soprano, Alto, Tenor and Bass (SATB), each with its own range of fundamental frequencies (F$0$s). The task of source separation for this choral setting entails separating the SATB mixture into the constituent parts. Source separation for musical mixtures is well studied and many deep learning based methodologies have been proposed for the same. However, most of the research has been foc…
Queen Mary University of London（伦敦玛丽女王大学）；Holonic Systems Oy.（霍洛尼克系统公司）
Smart cities are urban areas with sensor networks that collect data used towards efficient management. As a source of ubiquitous data, smart city initiatives present opportunities to enhance inhabitants' urban awareness. However, making sense of smart city data is challenging and there is a gap between available data and end-user applications. Sonification emerges as a promising method for the interpretation of smart city data and the production of novel musical experiences. In this paper, we first present the smart city paradigm. We then cover the topics of ubiquitous and mobile music, follow…
University of Oxford（牛津大学）；Institute for New Economic Thinking, University of Oxford（牛津大学新经济思维研究所）；Institute for Applied Mathematics and Systems (IIMAS), UNAM（墨西哥国立自治大学应用数学与系统研究所（IIMAS））；Fitzwilliam College, University of Cambridge（剑桥大学菲茨威廉学院）
There has been an everlasting discussion around the concept of form in music. This work is motivated by such debate by using a complex systems framework in which we study the form as an emergent property of rhythm. Such a framework corresponds with the traditional notion of musical form and allows us to generalize this concept to more general shapes and structures in music. We develop the three following metrics of the rhythmic complexity of a musical piece and its parts: 1) the rhythmic heterogeneity, based on the permutation entropy, where high values indicate a wide variety of rhythmic patt…
