Recent advances in text-to-music generation models have opened new avenues in musical creativity. However, music generation usually involves iterative refinements, and how to edit the generated music remains a significant challenge. This paper introduces a novel approach to the editing of music generated by such models, enabling the modification of specific attributes, such as genre, mood and instrument, while maintaining other aspects unchanged. Our method transforms text editing to \textit{latent space manipulation} while adding an extra constraint to enforce consistency. It seamlessly integrates with existing pretrained text-to-music diffusion models without requiring additional training. Experimental results demonstrate superior performance over both zero-shot and certain supervised baselines in style and timbre transfer evaluations. Additionally, we showcase the practical applicability of our approach in real-world music editing scenarios.
While deep generative models have empowered music generation, it remains a challenging and under-explored problem to edit an existing musical piece at fine granularity. In this paper, we propose SDMuse, a unified Stochastic Differential Music editing and generation framework, which can not only compose a whole musical piece from scratch, but also modify existing musical pieces in many ways, such as combination, continuation, inpainting, and style transferring. The proposed SDMuse follows a two-stage pipeline to achieve music generation and editing on top of a hybrid representation including pi…
The subjective evaluation of music generation techniques has been mostly done with questionnaire-based listening tests while ignoring the perspectives from music composition, arrangement, and soundtrack editing. In this paper, we propose an editing test to evaluate users' editing experience of music generation models in a systematic way. To do this, we design a new music style transfer model combining the non-chronological inference architecture, autoregressive models and the Transformer, which serves as an improvement from the baseline model on the same style transfer task. Then, we compare t…
BeatEdit: Symbolic Music Generation as Explicit Editing
School of Future Technology South China University of Technology Guangzhou China（未来技术学院 华南理工大学 广州 中国）；South China University of Technology（华南理工大学）；Nanjing University（南京大学）
Microsoft Research Asia（微软亚洲研究院）；Dalian University of Technology（大连理工大学）；Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）；Nanjing University（南京大学）
University of Alicante（阿利坎特大学）；TU Wien（维也纳工业大学）；Queen Mary University of London（伦敦玛丽女王大学）
The International Workshop on Reading Music Systems (WoRMS) is a workshop that tries to connect researchers who develop systems for reading music, such as in the field of Optical Music Recognition, with other researchers and practitioners that could benefit from such systems, like librarians or musicologists. The relevant topics of interest for the workshop include, but are not limited to: Music reading systems; Optical music recognition; Datasets and performance evaluation; Image processing on music scores; Writer identification; Authoring, editing, storing and presentation systems for music …
MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Massachusetts Institute of Technology（麻省理工学院）
In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments. We first identify two key intermediate representations for a successful video to music generator: body keypoints from videos and MIDI events from audio recordings. We then formulate music generation from videos as a motion-to-MIDI translation problem. We present a Graph$-$Transformer framework that can accurately predict MIDI event sequences in accordance with the body movements. The MIDI event can then be converted to realistic music using an …
MusicGen-Stem: Multi-stem music generation and edition through autoregressive modeling
Meta；Sorbonne Univ.（索邦大学）；UMR STMS IRCAM-CNRS；Univ. de Lorraine（洛林大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Loria（洛里亚研究所）；Hebrew Univ. of Jerusalem（耶路撒冷希伯来大学）
Many social media users prefer consuming content in the form of videos rather than text. However, in order for content creators to produce videos with a high click-through rate, much editing is needed to match the footage to the music. This posts additional challenges for more amateur video makers. Therefore, we propose a novel attention-based model VMT (Video-Music Transformer) that automatically generates piano scores from video frames. Using music generated from models also prevent potential copyright infringements that often come with using existing music. To the best of our knowledge, the…
Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation and Editing via Content-based Controls
New York University Shanghai（纽约大学上海分校）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Queen Mary University of London（伦敦大学玛丽皇后学院）
Some generative models for sequences such as music and text allow us to edit only subsequences, given surrounding context sequences, which plays an important part in steering generation interactively. However, editing subsequences mainly involves randomly resampling subsequences from a possible generation space. We propose a contextual latent space model (CLSM) in order for users to be able to explore subsequence generation with a sense of direction in the generation space, e.g., interpolation, as well as exploring variations -- semantically similar possible subsequences. A context-informed pr…
YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality
HKUST（香港科技大学）；M-A-P（M-A-P（多模态艺术投影））；New York University（纽约大学）；Stanford University（斯坦福大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；ACE Studio（ACE工作室）；HKGAI（香港生成式人工智能研究院）
AnyEdit: A Unified Framework for Speech and Singing Voice Editing with Real-World Environmental Consistency
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Institute of Media Technology and Experience Design, Huawei（华为媒体技术与体验设计研究院）；City University of Macau（澳门城市大学）；Amphion Technology Co., Ltd.（声智科技有限公司）
Mila, Quebec Artificial Intelligence Institute, Université de Montréal（魁北克人工智能研究所，蒙特利尔大学）；MIT - Massachusetts Institute of Technology（麻省理工学院）；Canada CIFAR AI Chair（加拿大 CIFAR 人工智能 chair）
Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generation
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；ByteDance Seed（字节跳动种子）；Shenzhen Loop Area Institute（深圳河套学院）；City University of Macau（澳门城市大学）；Amphion Technology Co., Ltd.（Amphion科技有限公司）
Dance is an important human art form, but creating new dances can be difficult and time-consuming. In this work, we introduce Editable Dance GEneration (EDGE), a state-of-the-art method for editable dance generation that is capable of creating realistic, physically-plausible dances while remaining faithful to the input music. EDGE uses a transformer-based diffusion model paired with Jukebox, a strong music feature extractor, and confers powerful editing capabilities well-suited to dance, including joint-wise conditioning, and in-betweening. We introduce a new metric for physical plausibility, …
SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor
The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（香港中文大学（深圳））；Tencent AI Lab（腾讯AI实验室）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Nanjing University（南京大学）
DirectAudioEdit: Inversion-Free Text-Guided Audio Editing via Diffusion Prediction Contrast
School of Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程学院）；Kunming University of Science and Technology（昆明理工大学）；NiuTrans Research, Shenyang, China（新译研究）
