Making a slight mistake during live music performance can easily be spotted by an astute listener, even if the performance is an improvisation or an unfamiliar piece. An example might be a highly dissonant chord played by mistake in a classical-era sonata, or a sudden off-key note in a recurring motif. The problem of identifying and correcting such errors can be approached with artificial intelligence -- if a trained human can easily do it, maybe a computer can be trained to spot the errors quickly and just as accurately. The ability to identify and auto-correct errors in real-time would be not only extremely useful to performing musicians, but also a valuable asset for producers, allowing much fewer overdubs and re-recording of takes due to small imperfections. This paper examines state-of-the-art solutions to related problems and explores novel solutions for music error detection and correction, focusing on their real-time applicability. The explored approaches consider error detection through music context and theory, as well as supervised learning models with no predefined musical information or rules, trained on appropriate datasets. Focusing purely on correcting musical errors, the presented solutions operate on a high-level representation of the audio (MIDI) instead of the raw audio domain, taking input from an electronic instrument (MIDI keyboard/piano) and altering it when needed before it is sent to the sampler. This work proposes multiple general recurrent neural network designs for real-time error correction and performance aid for MIDI instruments, discusses the results, limitations, and possible future improvements. It also emphasizes on making the research results easily accessible to the end user - music enthusiasts, producers and performers -- by using the latest artificial intelligence platforms and tools.
We introduce MIDI-VAE, a neural network model based on Variational Autoencoders that is capable of handling polyphonic music with multiple instrument tracks, as well as modeling the dynamics of music by incorporating note durations and velocities. We show that MIDI-VAE can perform style transfer on symbolic music by automatically changing pitches, dynamics and instruments of a music piece from, e.g., a Classical to a Jazz style. We evaluate the efficacy of the style transfer by training separate style validation classifiers. Our model can also interpolate between short pieces of music, produce…
In this paper, we introduce a simple method that can separate arbitrary musical instruments from an audio mixture. Given an unaligned MIDI transcription for a target instrument from an input mixture, we synthesize new mixtures from the midi transcription that sound similar to the mixture to be separated. This lets us create a labeled training set to train a network on the specific bespoke task. When this model applied to the original mixture, we demonstrate that this method can: 1) successfully separate out the desired instrument with access to only unaligned MIDI, 2) separate arbitrary instru…
Snap Inc.（Snap公司）；University of California, Berkeley（加州大学伯克利分校）；University of California, Los Angeles（加州大学洛杉矶分校）
In this paper, we propose a recurrent neural network (RNN)-based MIDI music composition machine that is able to learn musical knowledge from existing Beatles' songs and generate music in the style of the Beatles with little human intervention. In the learning stage, a sequence of stylistically uniform, multiple-channel music samples was modeled by a RNN. In the composition stage, a short clip of randomly-generated music was used as a seed for the RNN to start music score prediction. To form structured music, segments of generated music from different seeds were concatenated together. To improv…
The Accordiatron is a new MIDI controller for real-time performance based on the paradigm of a conventional squeeze box or concertina. It translates the gestures of a performer to the standard communication protocol of MIDI, allowing for flexible mappings of performance data to sonic parameters. When used in conjunction with a realtime signal processing environment, the Accordiatron becomes an expressive, versatile musical instrument. A combination of sensory outputs providing both discrete and continuous data gives the subtle expressiveness and control necessary for interactive music.
Just Another Media Laboratory (JAM Lab)（贾斯特另一媒体实验室（JAM实验室））；GITAM University（甘地理工与管理大学）
Generating music is an interesting and challenging problem in the field of machine learning. Mimicking human creativity has been popular in recent years, especially in the field of computer vision and image processing. With the advent of GANs, it is possible to generate new similar images, based on trained data. But this cannot be done for music similarly, as music has an extra temporal dimension. So it is necessary to understand how music is represented in digital form. When building models that perform this generative task, the learning and generation part is done in some high-level represen…
University of Sussex（萨塞克斯大学）；Singapore University of Technology and Design（新加坡科技设计大学）
We present a Python library, called Midi Miner, that can calculate tonal tension and classify different tracks. MIDI (Music Instrument Digital Interface) is a hardware and software standard for communicating musical events between digital music devices. It is often used for tasks such as music representation, communication between devices, and even music generation [5]. Tension is an essential element of the music listening experience, which can come from a number of musical features including timbre, loudness and harmony [3]. Midi Miner provides a Python implementation for the tonal tension m…
An ideal music synthesizer should be both interactive and expressive, generating high-fidelity audio in realtime for arbitrary combinations of instruments and notes. Recent neural synthesizers have exhibited a tradeoff between domain-specific models that offer detailed control of only specific instruments, or raw waveform models that can train on any music but with minimal control and slow generation. In this work, we focus on a middle ground of neural synthesizers that can generate audio from MIDI sequences with arbitrary combinations of instruments in realtime. This enables training on a wid…
Symbolic music datasets are important for music information retrieval and musical analysis. However, there is a lack of large-scale symbolic datasets for classical piano music. In this article, we create a GiantMIDI-Piano (GP) dataset containing 38,700,838 transcribed notes and 10,855 unique solo piano works composed by 2,786 composers. We extract the names of music works and the names of composers from the International Music Score Library Project (IMSLP). We search and download their corresponding audio recordings from the internet. We further create a curated subset containing 7,236 works c…
MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Massachusetts Institute of Technology（麻省理工学院）
In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments. We first identify two key intermediate representations for a successful video to music generator: body keypoints from videos and MIDI events from audio recordings. We then formulate music generation from videos as a motion-to-MIDI translation problem. We present a Graph$-$Transformer framework that can accurately predict MIDI event sequences in accordance with the body movements. The MIDI event can then be converted to realistic music using an …
Using open-source and creative coding frameworks, a teamof artist-engineers from Portland Community College work-ing with artists who experience Intellectual/Developmentaldisabilities prototyped an ensemble of adapted instrumentsand synthesizers that facilitate real-time in-key collabora-tion. The instruments employ a variety of sensors, send-ing the resulting musical controls to software sound gener-ators via MIDI. Careful consideration was given to the bal-ance between freedom of expression, and curating the pos-sible sonic outcomes as adaptation. Evaluation of adaptedinstrument design may d…
This study focuses on the perception of music performances when contextual factors, such as room acoustics and instrument, change. We propose to distinguish the concept of "performance" from the one of "interpretation", which expresses the "artistic intention". Towards assessing this distinction, we carried out an experimental evaluation where 91 subjects were invited to listen to various audio recordings created by resynthesizing MIDI data obtained through Automatic Music Transcription (AMT) systems and a sensorized acoustic piano. During the resynthesis, we simulated different contexts and a…
New York University Shanghai（纽约大学上海校区）；ByteDance（字节跳动）
We propose a unified model for three inter-related tasks: 1) to \textit{separate} individual sound sources from a mixed music audio, 2) to \textit{transcribe} each sound source to MIDI notes, and 3) to\textit{ synthesize} new pieces based on the timbre of separated sources. The model is inspired by the fact that when humans listen to music, our minds can not only separate the sounds of different instruments, but also at the same time perceive high-level representations such as score and timbre. To mirror such capability computationally, we designed a pitch-timbre disentanglement module based o…
Peking University（北京大学）；Wangxuan Institute of Computer Technology, PKU（北京大学王选计算机研究所）；China Conservatory of Music（中国音乐学院）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Unlike melody extraction and other aspects of music transcription, research on playing technique detection is still in its early stages. Compared to existing work mostly focused on playing technique detection for individual single notes, we propose a general end-to-end method based on Sound Event Detection by FCN for musical instrument playing technique detection. In our case, we choose Erhu, a well-known Chinese bowed-stringed instrument, to experiment with our method. Because of the limitation of FCN, we present an algorithm to detect on variable length audio. The effectiveness of the propos…
Performance Conditioning for Diffusion-Based Multi-Instrument Music Synthesis
International Audio Laboratories Erlangen（埃尔朗根国际音频实验室）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Fraunhofer Institute for Integrated Circuits IIS（弗劳恩霍夫集成电路研究所IIS）
We propose a novel system that takes as an input body movements of a musician playing a musical instrument and generates music in an unsupervised setting. Learning to generate multi-instrumental music from videos without labeling the instruments is a challenging problem. To achieve the transformation, we built a pipeline named 'Multi-instrumentalistNet' (MI Net). At its base, the pipeline learns a discrete latent representation of various instruments music from log-spectrogram using a Vector Quantized Variational Autoencoder (VQ-VAE) with multi-band residual blocks. The pipeline is then traine…
Queen Mary University of London（伦敦玛丽女王大学）；Indian Institute of Technology Kharagpur（印度理工学院哈拉格布尔分校）；National Taiwan University（国立台湾大学）；Dadabots（达达机器人）
Recently, symbolic music generation with deep learning techniques has witnessed steady improvements. Most works on this topic focus on MIDI representations, but less attention has been paid to symbolic music generation using guitar tablatures (tabs) which can be used to encode multiple instruments. Tabs include information on expressive techniques and fingerings for fretted string instruments in addition to rhythm and pitch. In this work, we use the DadaGP dataset for guitar tab music generation, a corpus of over 26k songs in GuitarPro and token formats. We introduce methods to condition a Tra…
Facebook AI Research（脸书人工智能研究院）；INRIA（法国国家信息与自动化研究所）；ENS（巴黎高等师范学院）；PSL Research University（巴黎文理研究大学）；LSCP（语言认知与计算实验室）；EHESS（法国高等社会科学研究院）；CNRS（法国国家科学研究中心）；École Normale Supérieure（巴黎高等师范学院）
Recent progress in deep learning for audio synthesis opens the way to models that directly produce the waveform, shifting away from the traditional paradigm of relying on vocoders or MIDI synthesizers for speech or music generation. Despite their successes, current state-of-the-art neural audio synthesizers such as WaveNet and SampleRNN suffer from prohibitive training and inference times because they are based on autoregressive models that generate audio samples one at a time at a rate of 16kHz. In this work, we study the more computationally efficient alternative of generating the waveform f…
Reduction of unwanted environmental noises is an important feature of today's hearing aids (HA), which is why noise reduction is nowadays included in almost every commercially available device. The majority of these algorithms, however, is restricted to the reduction of stationary noises. In this work, we propose a denoising approach based on a three hidden layer fully connected deep learning network that aims to predict a Wiener filtering gain with an asymmetric input context, enabling real-time applications with high constraints on signal delay. The approach is employing a hearing instrument…
