针对符号音乐数据稀缺问题，本文利用预训练 MIR 模型从音频自动提取符号事件训练生成模型，并提出 SymPAC，通过提示小节和 FSM 受约束生成提升可控性。
摘要：
符号音乐生成任务的进展可能落后于音频和文本生成等其他任务，部分原因在于符号训练数据的稀缺。本文中，我们利用规模更大的音频音乐数据，应用预训练的 MIR 模型（用于转谱、节拍跟踪、结构分析等）提取符号事件，并将其编码为 token 序列。据我们所知，本工作首次证明了仅依靠自动转谱音频数据训练符号生成模型的可行性。此外，为增强所训练模型的可控性，我们提出 SymPAC（带提示与受约束生成的符号音乐语言模型），其突出特点是在编码中使用（a）提示小节，以及在推理阶段使用（b）一种称为通过有限状态机（FSM）进行受约束生成的技术。我们展示了该方法的灵活性与可控性，这对于让音乐 AI 对创作者和用户有用可能至关重要。
英文摘要：
Progress in the task of symbolic music generation may be lagging behind other tasks like audio and text generation, in part because of the scarcity of symbolic training data. In this paper, we leverage the greater scale of audio music data by applying pre-trained MIR models (for transcription, beat tracking, structure analysis, etc.) to extract symbolic events and encode them into token sequences. To the best of our knowledge, this work is the first to demonstrate the feasibility of training symbolic generation models solely from auto-transcribed audio data. Furthermore, to enhance the controllability of the trained model, we introduce SymPAC (Symbolic Music Language Model with Prompting And Constrained Generation), which is distinguished by using (a) prompt bars in encoding and (b) a technique called Constrained Generation via Finite State Machines (FSMs) during inference time. We show the flexibility and controllability of this approach, which may be critical in making music AI useful to creators and users.
Beijing University of Posts and Telecommunications（北京邮电大学）
Music generation has always been a hot topic. When discussing symbolic music, melody or harmonies are usually seen as the only generating targets. But in fact, playing techniques are also quite an important part of the music. In this paper, we discuss the playing techniques generation problem by seeing it as a tagging problem. We propose a model that can use both the current data and external knowledge. Experiments were carried out by applying the proposed model in Chinese bamboo flute music, and results show that our method can make generated music more lively.
In this article, we present musicaiz, an object-oriented library for analyzing, generating and evaluating symbolic music. The submodules of the package allow the user to create symbolic music data from scratch, build algorithms to analyze symbolic music, encode MIDI data as tokens to train deep learning sequence models, modify existing music data and evaluate music generation systems. The evaluation submodule builds on previous work to objectively measure music generation systems and to be able to reproduce the results of music generation models. The library is publicly available online. We en…
Modelling human perception of musical similarity is critical for the evaluation of generative music systems, musicological research, and many Music Information Retrieval tasks. Although human similarity judgments are the gold standard, computational analysis is often preferable, since results are often easier to reproduce, and computational methods are much more scalable. Moreover, computation based approaches can be calculated quickly and on demand, which is a prerequisite for use with an online system. We propose StyleRank, a method to measure the similarity between a MIDI file and an arbitr…
SMUG-Explain: A Framework for Symbolic Music Graph Explanations
Institute of Computational Perception, Johannes Kepler University Linz（约翰内斯·开普勒大学林茨分校计算感知研究所）；LIT AI Lab, Linz Institute of Technology（林茨理工学院LIT人工智能实验室）
Music is a repetition of patterns and rhythms. It can be composed by repeating a certain number of bars in a structured way. In this paper, the objective is to generate a loop of 8 bars that can be used as a building block of music. Even considering musical diversity, we assume that music patterns familiar to humans can be defined in a finite set. With explicit rules to extract loops from music, we found that discrete representations are sufficient to model symbolic music sequences. Among VAE family, musical properties from VQ-VAE are better observed rather than other models. Further, to empha…
8+8=4: Formalizing Time Units to Handle Symbolic Music Durations
Johannes Kepler University Linz（林茨约翰内斯·开普勒大学）；CNAM Paris（巴黎国立工艺学院）；Nagoya University（名古屋大学）；Asia University（亚细亚大学）；Linz Institute of Technology（林茨技术研究所）
Symbolic music understanding, which refers to the understanding of music from the symbolic data (e.g., MIDI format, but not audio), covers many music applications such as genre classification, emotion classification, and music pieces matching. While good music representations are beneficial for these applications, the lack of training data hinders representation learning. Inspired by the success of pre-training models in natural language processing, in this paper, we develop MusicBERT, a large-scale pre-trained model for music understanding. To this end, we construct a large-scale symbolic mus…
NotaGen: Advancing Musicality in Symbolic Music Generation with Large Language Model Training Paradigms
Central Conservatory of Music（中央音乐学院）；University of Rochester（罗切斯特大学）；Beijing Flowingtech Ltd.（北京流音科技有限公司）；Beihang University（北京航空航天大学）；Tsinghua University（清华大学）
LG AI Research（LG人工智能研究院）；University of Ulsan（蔚山大学）
Similar to colorization in computer vision, instrument separation is to assign instrument labels (e.g. piano, guitar...) to notes from unlabeled mixtures which contain only performance information. To address the problem, we adopt diffusion models and explicitly guide them to preserve consistency between mixtures and music. The quantitative results show that our proposed model can generate high-fidelity samples for multitrack symbolic music with creativity.
Partitura is a lightweight Python package for handling symbolic musical information. It provides easy access to features commonly used in music information retrieval tasks, like note arrays (lists of timed pitched events) and 2D piano roll matrices, as well as other score elements such as time and key signatures, performance directives, and repeat structures. Partitura can load musical scores (in MEI, MusicXML, Kern, and MIDI formats), MIDI performances, and score-to-performance alignments. The package includes some tools for music analysis, such as automatic pitch spelling, key signature iden…
Institute of Computational Perception, JKU Linz（林茨约翰·开普勒大学计算感知研究所）；Sony Computer Science Laboratories (CSL)（索尼计算机科学实验室）
Many music theoretical constructs (such as scale types, modes, cadences, and chord types) are defined in terms of pitch intervals---relative distances between pitches. Therefore, when computer models are employed in music tasks, it can be useful to operate on interval representations rather than on the raw musical surface. Moreover, interval representations are transposition-invariant, valuable for tasks like audio alignment, cover song detection and music structure analysis. We employ a gated autoencoder to learn fixed-length, invertible and transposition-invariant interval representations fr…
Music Structure Analysis is an open research task in Music Information Retrieval (MIR). In the past, there have been several works that attempt to segment music into the audio and symbolic domains, however, the identification and segmentation of the music structure at different levels is still an open research problem in this area. In this work we propose three methods, two of which are novel graph-based algorithms that aim to segment symbolic music by its form or structure: Norm, G-PELT and G-Window. We performed an ablation study with two public datasets that have different forms or structur…
Nankai University（南开大学）；National University of Singapore（新加坡国立大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；Tsinghua University（清华大学）
Definitive embeddings remain a fundamental challenge of computational musicology for symbolic music in deep learning today. Analogous to natural language, music can be modeled as a sequence of tokens. This motivates the majority of existing solutions to explore the utilization of word embedding models to build music embeddings. However, music differs from natural languages in two key aspects: (1) musical token is multi-faceted -- it comprises of pitch, rhythm and dynamics information; and (2) musical context is two-dimensional -- each musical token is dependent on both melodic and harmonic con…
LG AI Research（LG人工智能研究院）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Since most of music has repetitive structures from motifs to phrases, repeating musical ideas can be a basic operation for music composition. The basic block that we focus on is conceptualized as loops which are essential ingredients of music. Furthermore, meaningful note patterns can be formed in a finite space, so it is sufficient to represent them with combinations of discrete symbols as done in other domains. In this work, we propose symbolic music loop generation via learning discrete representations. We first extract loops from MIDI datasets using a loop detector and then learn an autore…
A key aspect of machine learning models lies in their ability to learn efficient intermediate features. However, the input representation plays a crucial role in this process, and polyphonic musical scores remain a particularly complex type of information. In this paper, we introduce a novel representation of symbolic music data, which transforms a polyphonic score into a continuous signal. We evaluate the ability to learn meaningful features from this representation from a musical point of view. Hence, we introduce an evaluation method relying on principled generation of synthetic data. Final…
Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey
Univ. Lille（里尔大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Centrale Lille（里尔中央理工学院）；Univ. Bordeaux（波尔多大学）；Bordeaux INP（波尔多综合理工学院）；Singapore University of Technology and Design（新加坡科技设计大学）
Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization
Sound and Music Computing Lab, School of Computing, NUS（新加坡国立大学计算机学院声音与音乐计算实验室）；Courant Institute of Mathematical Sciences, New York University（纽约大学应用数学科学学院）；Music X Lab, MBZUAI（MBZUAI音乐X实验室）
Symbolic Music Generation with Non-Differentiable Rule Guided Diffusion
California Institute of Technology（加州理工学院）；Rensselaer Polytechnic Institute（伦斯勒理工学院）；NVIDIA（英伟达公司）；Dalhousie University（达尔豪斯大学）；Vector Institute（矢量研究所）
Symbolic Music Generation relies on the contextual representation capabilities of the generative model, where the most prevalent approach is the Transformer-based model. The learning of musical context is also related to the structural elements in music, i.e. intro, verse, and chorus, which are currently overlooked by the research community. In this paper, we propose a hierarchical Transformer model to learn multi-scale contexts in music. In the encoding phase, we first designed a Fragment Scope Localization layer to syncopate the music into chords and sections. Then, we use a multi-scale atte…
Structure perception is a fundamental aspect of music cognition in humans. Historically, the hierarchical organization of music into structures served as a narrative device for conveying meaning, creating expectancy, and evoking emotions in the listener. Thereby, musical structures play an essential role in music composition, as they shape the musical discourse through which the composer organises his ideas. In this paper, we present a novel music segmentation method, pitchclass2vec, based on symbolic chord annotations, which are embedded into continuous vector representations using both natur…
