Generating music with deep neural networks has been an area of active research in recent years. While the quality of generated samples has been steadily increasing, most methods are only able to exert minimal control over the generated sequence, if any. We propose the self-supervised description-to-sequence task, which allows for fine-grained controllable generation on a global level. We do so by extracting high-level features about the target sequence and learning the conditional distribution of sequences given the corresponding high-level description in a sequence-to-sequence modelling setup. We train FIGARO (FIne-grained music Generation via Attention-based, RObust control) by applying description-to-sequence modelling to symbolic music. By combining learned high level features with domain knowledge, which acts as a strong inductive bias, the model achieves state-of-the-art results in controllable symbolic music generation and generalizes well beyond the training distribution.
This study focuses on the perception of music performances when contextual factors, such as room acoustics and instrument, change. We propose to distinguish the concept of "performance" from the one of "interpretation", which expresses the "artistic intention". Towards assessing this distinction, we carried out an experimental evaluation where 91 subjects were invited to listen to various audio recordings created by resynthesizing MIDI data obtained through Automatic Music Transcription (AMT) systems and a sensorized acoustic piano. During the resynthesis, we simulated different contexts and a…
Télécom Paris（巴黎电信学院）；Institut Polytechnique de Paris（巴黎综合理工学院）
Research on style transfer and domain translation has clearly demonstrated the ability of deep learning-based algorithms to manipulate images in terms of artistic style. More recently, several attempts have been made to extend such approaches to music (both symbolic and audio) in order to enable transforming musical style in a similar manner. In this study, we focus on symbolic music with the goal of altering the 'style' of a piece while keeping its original 'content'. As opposed to the current methods, which are inherently restricted to be unsupervised due to the lack of 'aligned' data (i.e. …
Key Laboratory of Noise and Vibration Research, Institute of Acoustics, Chinese Academy of Sciences（中国科学院声学研究所噪声与振动研究重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；State Key Laboratory of Media Convergence and Communication, Communication University of China（中国传媒大学媒体融合与传播国家重点实验室）；School of Electronics and Communication Engineering, Guangzhou University（广州大学电子与通信工程学院）；National Environmental Protection Engineering and Technology Center for Road Traffic Noise Control（国家环境保护道路交通噪声控制工程技术中心）
With the development of deep neural networks, automatic music composition has made great progress. Although emotional music can evoke listeners' different emotions and it is important for artistic expression, only few researches have focused on generating emotional music. This paper presents EmotionBox -an music-element-driven emotional music generator that is capable of composing music given a specific emotion, where this model does not require a music dataset labeled with emotions. Instead, pitch histogram and note density are extracted as features that represent mode and tempo respectively …
Central Conservatory of Music（中央音乐学院）；Tsinghua University（清华大学）
In natural language processing (NLP), the semantic similarity task requires large-scale, high-quality human-annotated labels for fine-tuning or evaluation. By contrast, in cases of music similarity, such labels are expensive to collect and largely dependent on the annotator's artistic preferences. Recent research has demonstrated that embedding calibration technique can greatly increase semantic similarity performance of the pre-trained language model without fine-tuning. However, it is yet unknown which calibration method is the best and how much performance improvement can be achieved. To ad…
Volkswagen Group（大众集团）；ArgoAI；St. Pölten University of Applied Sciences（圣帕尔滕应用科学大学）；St Joost School of Art and Design（圣约斯特艺术与设计学院）
The use of machine learning in artistic music generation leads to controversial discussions of the quality of art, for which objective quantification is nonsensical. We therefore consider a music-generating algorithm as a counterpart to a human musician, in a setting where reciprocal interplay is to lead to new experiences, both for the musician and the audience. To obtain this behaviour, we resort to the framework of recurrent Variational Auto-Encoders (VAE) and learn to generate music, seeded by a human musician. In the learned model, we generate novel musical sequences by interpolation in l…
Since the introduction of deep learning, researchers have proposed content generation systems using deep learning and proved that they are competent to generate convincing content and artistic output, including music. However, one can argue that these deep learning-based systems imitate and reproduce the patterns inherent within what humans have created, instead of generating something new and creative. This paper focuses on music generation, especially rhythm patterns of electronic dance music, and discusses if we can use deep learning to generate novel rhythms, interesting patterns not found…
Workflow-Based Evaluation of Music Generation Systems
Department of Computer Science UiT, The Arctic University of Tromsø（计算机科学系，UiT极地大学）；Department of Music NTNU, Norwegian University of Science and Technology（音乐系，NTNU挪威科学与技术大学）
Dance-to-Music Generation with Encoder-based Textual Inversion
Institute of Automation, CAS（中国科学院自动化研究所）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；University of Chinese Academy of Sciences（中国科学院大学）；Kuaishou Technology（快手科技）；University of Konstanz（康斯坦茨大学）；National Cheng-Kung University（成功大学）
Center for Computer Research in Music and Acoustics (CCRMA), Stanford University（斯坦福大学音乐与声学计算机研究中心）；Stanford Artificial Intelligence Laboratory（斯坦福人工智能实验室）；Design Program, Department of Mechanical Engineering, Stanford University（斯坦福大学机械工程系设计项目）
We propose a system that learns from artistic pairings of music and corresponding album cover art. The goal is to 'translate' paintings into music and, in further stages of development, the converse. We aim to deploy this system as an artistic tool for real time 'translations' between musicians and painters. The system's outputs serve as elements to be employed in a joint live performance of music and painting, or as generative material to be used by the artists as inspiration for their improvisation.
This paper will present observations on the design, artistic, and human factors of creating digital music controllers. Specific projects will be presented, and a set of design principles will be supported from those examples.
Melody or Machine: Detecting Synthetic Music with Dual-Stream Contrastive Learning
Indraprastha Institute of Information Technology Delhi (IIIT-Delhi)（印度理工学院德里分校）；Manipal University Jaipur（曼海姆大学斋普尔）；Netaji Subhas University of Technology (NSUT)（尼赫鲁大学技术学院）
Performance RNN is a machine-learning system designed primarily for the generation of solo piano performances using an event-based (rather than audio) representation. More specifically, Performance RNN is a long short-term memory (LSTM) based recurrent neural network that models polyphonic music with expressive timing and dynamics (Oore et al., 2018). The neural network uses a simple language model based on the Musical Instrument Digital Interface (MIDI) file format. Performance RNN is trained on the e-Piano Junior Competition Dataset (International Piano e-Competition, 2018), a collection of …
Queen Mary University of London（伦敦玛丽女王大学）；Vrije Universiteit Brussel（布鲁塞尔自由大学）
Music Performers have their own idiosyncratic way of interpreting a musical piece. A group of skilled performers playing the same piece of music would likely to inject their unique artistic styles in their performances. The variations of the tempo, timing, dynamics, articulation etc. from the actual notated music are what make the performers unique in their performances. This study presents a dataset consisting of four movements of Schubert's ``Sonata in B-flat major, D.960" performed by nine virtuoso pianists individually. We proposed and extracted a set of expressive features that are able t…
Creation of images using generative adversarial networks has been widely adapted into multi-modal regime with the advent of multi-modal representation models pre-trained on large corpus. Various modalities sharing a common representation space could be utilized to guide the generative models to create images from text or even from audio source. Departing from the previous methods that solely rely on either text or audio, we exploit the expressiveness of both modality. Based on the fusion of text and audio, we create video whose content is consistent with the distinct modalities that are provid…
School of Cyber Science and Engineering, Southeast University, Nanjing 210096, China（东南大学计算机科学与工程学院，南京210096，中国）；School of Computer Science and Engineering, Southeast University, Nanjing 210096, China（东南大学计算机科学与工程学院，南京210096，中国）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其交叉应用关键实验室（东南大学），教育部，中国）；Purple Mountain Laboratories, Nanjing 210000, China（紫金山实验室，南京210000，中国）；Engineering Research Center of Blockchain Application, Supervision And Management (Southeast University), Ministry of Education, China（区块链应用、监督与管理工程研究中心（东南大学），教育部，中国）
MG-Former: A Transformer-Based Framework for Music-Driven 3D Conducting Gesture Generation
Malou Tech Inc（Malou科技公司）；South-Central Minzu University（西南民族大学）；Beijing Jiaotong University（北京交通大学）；Fudan University（复旦大学）；Renmin University of China（中国人民大学）
Nowadays, technological advances have influenced all human activities, creating new dynamics and ways of communication. In this context, some artists have incorporated these advances in their creative process, giving rise to unique aesthetic expressions referred to in the literature as Generative Art, which is characterized by assigning part of the creative process to a system that acts with certain autonomy (Galanter, 2003). This research work introduces a computational system for creating generative art using a Brain-Computer Interface (BCI) which portrays the user's brain activity in a digi…
MuDiT & MuSiT: Alignment with Colloquial Expression in Description-to-Song Generation
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Caichong Artificial Intelligence Research, DuiNiuTanQin Co., Ltd.（追牛探琴有限公司彩创人工智能研究院）
