Fundamental frequency (f0) estimation from polyphonic music includes the tasks of multiple-f0, melody, vocal, and bass line estimation. Historically these problems have been approached separately, and only recently, using learning-based approaches. We present a multitask deep learning architecture that jointly estimates outputs for various tasks including multiple-f0, melody, vocal and bass line estimation, and is trained using a large, semi-automatically annotated dataset. We show that the multitask model outperforms its single-task counterparts, and explore the effect of various design decisions in our approach, and show that it performs better or at least competitively when compared against strong baseline methods.
Research Center for IT Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）；KKBOX Inc.（KKBOX公司）
For many music analysis problems, we need to know the presence of instruments for each time frame in a multi-instrument musical piece. However, such a frame-level instrument recognition task remains difficult, mainly due to the lack of labeled datasets. To address this issue, we present in this paper a large-scale dataset that contains synthetic polyphonic music with frame-level pitch and instrument labels. Moreover, we propose a simple yet novel network architecture to jointly predict the pitch and instrument for each frame. With this multitask learning method, the pitch information can be le…
This paper introduces the jazznet Dataset, a dataset of fundamental jazz piano music patterns for developing machine learning (ML) algorithms in music information retrieval (MIR). The dataset contains 162520 labeled piano patterns, including chords, arpeggios, scales, and chord progressions with their inversions, resulting in more than 26k hours of audio and a total size of 95GB. The paper explains the dataset's composition, creation, and generation, and presents an open-source Pattern Generator using a method called Distance-Based Pattern Structures (DBPS), which allows researchers to easily …
This paper explores a variety of models for frame-based music transcription, with an emphasis on the methods needed to reach state-of-the-art on human recordings. The translation-invariant network discussed in this paper, which combines a traditional filterbank with a convolutional neural network, was the top-performing model in the 2017 MIREX Multiple Fundamental Frequency Estimation evaluation. This class of models shares parameters in the log-frequency domain, which exploits the frequency invariance of music to reduce the number of model parameters and avoid overfitting to the training data…
This paper refers to the extraction of multiple fundamental frequencies (multiple F0) based on PYIN, an algorithm for extracting the fundamental frequency (F0) of monophonic music, and a trained convolutional neural networks (CNN) model, where a pitch salience function of the input signal is produced to estimate the multiple F0. The implementation of these two algorithms and their corresponding advantages and disadvantages are discussed in this article. Analysing the different performance of these two methods, PYIN is applied to supplement the F0 extracted from the trained CNN model to combine…
Current AI-generated music lacks fundamental principles of good compositional techniques. By narrowing down implementation issues both programmatically and musically, we can create a better understanding of what parameters are necessary for a generated composition nearly indistinguishable from that of a master composer.
We propose a novel pitch estimation technique called DeepF0, which leverages the available annotated data to directly learns from the raw audio in a data-driven manner. F0 estimation is important in various speech processing and music information retrieval applications. Existing deep learning models for pitch estimations have relatively limited learning capabilities due to their shallow receptive field. The proposed model addresses this issue by extending the receptive field of a network by introducing the dilated convolutional blocks into the network. The dilation factor increases the network…
Elastic overtones: an equal temperament 12 tone music system with "perfect" fifths
Instituto de Astronomía, Universidad Nacional Autónoma de México（墨西哥国立自治大学天文研究所）；Department of Physics, Loyola University Chicago（芝加哥洛约拉大学物理系）；School of Audio Engineering (SAE Institute Mexico)（音频工程学院（墨西哥SAE学院））
Speech coding forms a crucial element in speech communications. An important area concerning it lies in feature extraction which can be used for analyzing Hindustani Classical Music. An important feature in this respect is the fundamental frequency often referred to as the pitch. In this work, the terms pitch and its acoustical sensation, the frequency is used interchangeably. There exists numerous pitch detection algorithms which detect the main/ fundamental frequency in a given musical piece, but we have come up with a unique algorithm for pitch detection using the binning method as describe…
Lightweight Self-Supervised Detection of Fundamental Frequency and Accurate Probability of Voicing in Monophonic Music
Dept. of Computer Science, Columbia University, New York, US（计算机科学系，哥伦比亚大学，纽约，美国）；Dept. of Mechanical Engineering and Dept. of Electrical Engineering, Columbia University（机械工程系和电气工程系，哥伦比亚大学）；Recognition Technologies, Inc., New York, US（识别技术公司，纽约，美国）
Université Côte d’Azur（蔚蓝海岸大学）；CNRS（法国国家科学研究中心）；OCA（蔚蓝海岸天文台）；IRIT（图卢兹计算机科学研究所）；Université de Bretagne Sud（南布列塔尼大学）；IRISA（法国国家信息与自动化研究所-雷恩第一大学联合研究中心）；Aix-Marseille Université（艾克斯-马赛大学）；LIF（计算机科学实验室）
Many spectral unmixing methods rely on the non-negative decomposition of spectral data onto a dictionary of spectral templates. In particular, state-of-the-art music transcription systems decompose the spectrogram of the input signal onto a dictionary of representative note spectra. The typical measures of fit used to quantify the adequacy of the decomposition compare the data and template entries frequency-wise. As such, small displacements of energy from a frequency bin to another as well as variations of timber can disproportionally harm the fit. We address these issues by means of optimal …
Automatic Chord Estimation (ACE) is a fundamental task in Music Information Retrieval (MIR) and has applications in both music performance and MIR research. The task consists of segmenting a music recording or score and assigning a chord label to each segment. Although it has been a task in the annual benchmarking evaluation MIREX for over 10 years, ACE is not yet a solved problem, since performance has stagnated and modern systems have started to tune themselves to subjective training data. We propose DECIBEL, a new ACE system that exploits widely available MIDI and tab representations to imp…
Research Center for Information Technology Innovation, Academia Sinica（中央研究院信息技术创新研究中心）
Singing voice separation attempts to separate the vocal and instrumental parts of a music recording, which is a fundamental problem in music information retrieval. Recent work on singing voice separation has shown that the low-rank representation and informed separation approaches are both able to improve separation quality. However, low-rank optimizations are computationally inefficient due to the use of singular value decompositions. Therefore, in this paper, we propose a new linear-time algorithm called informed group-sparse representation, and use it to separate the vocals from music using…
Indian Institute of Technology, Kanpur（印度理工学院坎普尔分校）
Melody estimation or melody extraction refers to the extraction of the primary or fundamental dominant frequency in a melody. This sequence of frequencies obtained represents the pitch of the dominant melodic line from recorded music audio signals. The music signal may be monophonic or polyphonic. The melody extraction problem from audio signals gets complicated when we start dealing with polyphonic audio data. This is because in generalized audio signals,the sounds are highly correlated over both frequency and time domains. This complex overlap of many sounds, makes identification of predomin…
Frequency-Aware Self-Supervised Music Representation Learning
Spellbrush；Acoustic Lab, Department of Information and Communications Engineering (DICE), Aalto University（阿尔托大学信息与通信工程系声学实验室）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
A raga is a melodic structure with fixed notes and a set of rules characterizing a certain mood endorsed through performance. By a vadi swar is meant that note which plays the most significant role in expressing the raga. A samvadi swar similarly is the second most significant note. However, the determination of their significance has an element of subjectivity and hence we are motivated to find some truths through an objective analysis. The paper proposes a probabilistic method of note detection and demonstrates how the relative frequency (relative number of occurrences of the pitch) of the m…
University of Oldenburg（奥尔登堡大学）；Hörzentrum Oldenburg gGmbH（奥尔登堡听力中心）；Cluster of Excellence Hearing4all（Hearing4all 卓越集群）
Many speech and music analysis and processing schemes rely on an estimate of the fundamental frequency $f_0$ of periodic signal components. Most established schemes apply rather unspecific signal models such as sinusoidal models to the estimation problem, which may limit time resolution and estimation accuracy. This study proposes a novel time-domain locked-loop algorithm with low computational effort and low memory footprint for $f_0$ estimation. The loop control signal is directly derived from the input time signal, using a harmonic signal model. Theoretically, this allows for a noise-robust…
This paper presents a new method of singing voice analysis that performs mutually-dependent singing voice separation and vocal fundamental frequency (F0) estimation. Vocal F0 estimation is considered to become easier if singing voices can be separated from a music audio signal, and vocal F0 contours are useful for singing voice separation. This calls for an approach that improves the performance of each of these tasks by using the results of the other. The proposed method first performs robust principal component analysis (RPCA) for roughly extracting singing voices from a target music audio s…
