Developing open-source foundation models is essential for advancing research in music audio understanding and ensuring access to powerful, multipurpose representations for music information retrieval. We present OMAR-RQ, a model trained with self-supervision via masked token classification methodologies using a large-scale dataset with over 330,000 hours of music audio. We experiment with different input features and quantization options, and achieve state-of-the-art performance in music tagging, pitch estimation, chord recognition, beat tracking, segmentation, and difficulty estimation among open self-supervised models. We open-source our training and evaluation pipelines and model weights, available at https://github.com/mtg/omar-rq.
Frequency-Aware Self-Supervised Music Representation Learning
Spellbrush；Acoustic Lab, Department of Information and Communications Engineering (DICE), Aalto University（阿尔托大学信息与通信工程系声学实验室）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
Despite recent achievements of deep learning automatic music generation algorithms, few approaches have been proposed to evaluate whether a single-track music excerpt is composed by automatons or Homo sapiens. To tackle this problem, we apply a masked language model based on ALBERT for composers classification. The aim is to obtain a model that can suggest the probability a MIDI clip might be composed condition on the auto-generation hypothesis, and which is trained with only AI-composed single-track MIDI. In this paper, the amount of parameters is reduced, two methods on data augmentation are…
Neural network based architectures used for sound recognition are usually adapted from other application domains such as image recognition, which may not harness the time-frequency representation of a signal. The ConditionaL Neural Networks (CLNN) and its extension the Masked ConditionaL Neural Networks (MCLNN) are designed for multidimensional temporal signal recognition. The CLNN is trained over a window of frames to preserve the inter-frame relation, and the MCLNN enforces a systematic sparseness over the network's links that mimics a filterbank-like behavior. The masking operation induces …
Johns Hopkins University（约翰斯·霍普金斯大学）；University of Western Australia（西澳大利亚大学）
Lyrics recognition is an important task in music processing. Despite traditional algorithms such as the hybrid HMM- TDNN model achieving good performance, studies on applying end-to-end models and self-supervised learning (SSL) are limited. In this paper, we first establish an end-to-end baseline for lyrics recognition and then explore the performance of SSL models on lyrics recognition task. We evaluate a variety of upstream SSL models with different training methods (masked reconstruction, masked prediction, autoregressive reconstruction, and contrastive learning). Our end-to-end self-superv…
Department of Electronic Engineering University of York（约克大学电子工程系）；University of York（约克大学）
We present the ConditionaL Neural Network (CLNN) and the Masked ConditionaL Neural Network (MCLNN) designed for temporal signal recognition. The CLNN takes into consideration the temporal nature of the sound signal and the MCLNN extends upon the CLNN through a binary mask to preserve the spatial locality of the features and allows an automated exploration of the features combination analogous to hand-crafting the most relevant features for the recognition task. MCLNN has achieved competitive recognition accuracies on the GTZAN and the ISMIR2004 music datasets that surpass several state-of-the-…
MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
Hong Kong University of Science and Technology（香港科技大学）；University of Waterloo（滑铁卢大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；University of Manchester（曼彻斯特大学）；University of Sheffield（谢菲尔德大学）；Carnegie Mellon University（卡内基梅隆大学）；Queen Mary University of London（伦敦大学玛丽皇后学院）；Durham University（杜伦大学）；Google DeepMind（谷歌DeepMind）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；New York University（纽约大学）
SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor
The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（香港中文大学（深圳））；Tencent AI Lab（腾讯AI实验室）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Nanjing University（南京大学）
AaSP: Aliasing-aware Self-Supervised Pre-Training for Audio Spectrogram Transformers
Research & Development Center, Technology Division, Oki Electric Industry Co., Ltd.（oki电产业株式会社研发中心，技术部门）；Department of Data Science for Business Innovation, Chuo University（中央大学商务创新数据科学系）
