Centre for Digital Music, Queen Mary University of London(伦敦大学玛丽皇后学院数字音乐中心);School of Computer Science, Carnegie Mellon University(卡内基梅隆大学计算机学院);University of Sheffield(谢菲尔德大学);The University of Manchester(曼彻斯特大学);University of Auckland(奥克兰大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);New York University Shanghai(纽约大学上海分校);University of Tokyo(东京大学)
Yinghao Ma, Anders Øland, Anton Ragni, Bleiz MacSen Del Sette, Charalampos Saitis, Chris Donahue, Chenghua Lin, Christos Plachouras, Emmanouil Benetos, Elona Sh… 展开作者
Yinghao Ma, Anders Øland, Anton Ragni, Bleiz MacSen Del Sette, Charalampos Saitis, Chris Donahue, Chenghua Lin, Christos Plachouras, Emmanouil Benetos, Elona Shatri, Fabio Morreale, Ge Zhang, György Fazekas, Gus Xia, Huan Zhang, Ilaria Manco, Jiawen Huang, Julien Guinot, Liwei Lin, Luca Marinelli, Max W. Y. Lam, Megha Sharma, Qiuqiang Kong, Roger B. Dannenberg, Ruibin Yuan, Shangda Wu, Shih-Lun Wu, Shuqi Dai, Shun Lei, Shiyin Kang, Simon Dixon, Wenhu Chen, Wenhao Huang, Xingjian Du, Xingwei Qu, Xu Tan, Yizhi Li, Zeyue Tian, Zhiyong Wu, Zhizheng Wu, Ziyang Ma, Ziyu Wang
In recent years, foundation models (FMs) such as large language models (LLMs) and latent diffusion models (LDMs) have profoundly impacted diverse sectors, including music. This comprehensive review examines state-of-the-art (SOTA) pre-trained models and foundation models in music, spanning from representation learning, generative learning and multimodal learning. We first contextualise the significance of music in various industries and trace the evolution of AI in music. By delineating the modalities targeted by foundation models, we discover many of the music representations are underexplored in FM development. Then, emphasis is placed on the lack of versatility of previous methods on diverse music applications, along with the potential of FMs in music understanding, generation and medical application. By comprehensively exploring the details of the model pre-training paradigm, architectural choices, tokenisation, finetuning methodologies and controllability, we emphasise the important topics that should have been well explored, like instruction tuning and in-context learning, scaling law and emergent ability, as well as long-sequence modelling etc. A dedicated section presents insights into music agents, accompanied by a thorough analysis of datasets and evaluations essential for pre-training and downstream tasks. Finally, by underscoring the vital importance of ethical considerations, we advocate that following research on FM for music should focus more on such issues as interpretability, transparency, human responsibility, and copyright issues. The paper offers insights into future challenges and trends on FMs for music, aiming to shape the trajectory of human-AI collaboration in the music realm.
A Survey of Foundation Models for Music Understanding
School of Automation, Northwestern Polytechnical University(西北工业大学自动化学院);School of Physics and Information Technology, Shaanxi Normal University(陕西师范大学物理与信息技术学院)
Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey
Univ. Lille(里尔大学);CNRS(法国国家科学研究中心);Inria(法国国家信息与自动化研究所);Centrale Lille(里尔中央理工学院);Univ. Bordeaux(波尔多大学);Bordeaux INP(波尔多综合理工学院);Singapore University of Technology and Design(新加坡科技设计大学)
NOTA: Multimodal Music Notation Understanding for Visual Large Language Model
School of Computer Science, Wuhan University(武汉大学计算机学院);Key Laboratory of Archival Intelligent Development and Service, NAAC(国家档案学院档案智能发展与服务重点实验室);School of Information Management, Wuhan University(武汉大学信息管理学院)
BeatFM: Improving Beat Tracking with Pre-trained Music Foundation Model
School of Computer Science, Fudan University, Shanghai, China(复旦大学计算机科学学院);Naval Medical Center, PLA, China(中国人民解放军海军医疗中心);Graduate School of Informatics, Kyoto University, Kyoto, Japan(京都大学信息科学研究生院);Graduate School of Advanced Science and Engineering, Hiroshima University, Hiroshima, Japan(广岛大学先进科学与工程研究生院);Shanghai Key Laboratory of Intelligent Information Processing, Fudan University, Shanghai, China(上海智能信息处理重点实验室)
Shanghai Jiao Tong University(上海交通大学);Music Tech Lab, DynamiX(音乐科技实验室,DynamiX);Shanghai AI Laboratory(上海人工智能实验室);Beijing Film Academy(北京电影学院);Tsinghua University(清华大学);McGill University(麦吉尔大学);The Chinese University of Hong Kong(香港中文大学)
A Survey on Cross-Modal Interaction Between Music and Multimodal Data
MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China. E-mail: S. Li;School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100190, China. E-mail: F. Shen;Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China. E-mail: .
A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Innovation Center of Yangtze River Delta, Zhejiang University(浙江大学长三角创新中心)