A Survey of Foundation Models for Music Understanding
School of Automation, Northwestern Polytechnical University(西北工业大学自动化学院);School of Physics and Information Technology, Shaanxi Normal University(陕西师范大学物理与信息技术学院)
Wenjun Li, Ying Cai, Ziyang Wu, Wenyi Zhang, Yifan Chen, Rundong Qi, Mengqi Dong, Peigen Chen, Xiao Dong, Fenghao Shi, Lei Guo, Junwei Han, Bao Ge, Tianming Liu, Lin Gan, Tuo Zhang
Music is essential in daily life, fulfilling emotional and entertainment needs, and connecting us personally, socially, and culturally. A better understanding of music can enhance our emotions, cognitive skills, and cultural connections. The rapid advancement of artificial intelligence (AI) has introduced new ways to analyze music, aiming to replicate human understanding of music and provide related services. While the traditional models focused on audio features and simple tasks, the recent development of large language models (LLMs) and foundation models (FMs), which excel in various fields by integrating semantic information and demonstrating strong reasoning abilities, could capture complex musical features and patterns, integrate music with language and incorporate rich musical, emotional and psychological knowledge. Therefore, they have the potential in handling complex music understanding tasks from a semantic perspective, producing outputs closer to human perception. This work, to our best knowledge, is one of the early reviews of the intersection of AI techniques and music understanding. We investigated, analyzed, and tested recent large-scale music foundation models in respect of their music comprehension abilities. We also discussed their limitations and proposed possible future directions, offering insights for researchers in this field.
Centre for Digital Music, Queen Mary University of London(伦敦大学玛丽皇后学院数字音乐中心);School of Computer Science, Carnegie Mellon University(卡内基梅隆大学计算机学院);University of Sheffield(谢菲尔德大学);The University of Manchester(曼彻斯特大学);University of Auckland(奥克兰大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);New York University Shanghai(纽约大学上海分校);University of Tokyo(东京大学)
Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey
Univ. Lille(里尔大学);CNRS(法国国家科学研究中心);Inria(法国国家信息与自动化研究所);Centrale Lille(里尔中央理工学院);Univ. Bordeaux(波尔多大学);Bordeaux INP(波尔多综合理工学院);Singapore University of Technology and Design(新加坡科技设计大学)
NOTA: Multimodal Music Notation Understanding for Visual Large Language Model
School of Computer Science, Wuhan University(武汉大学计算机学院);Key Laboratory of Archival Intelligent Development and Service, NAAC(国家档案学院档案智能发展与服务重点实验室);School of Information Management, Wuhan University(武汉大学信息管理学院)
Shanghai Jiao Tong University(上海交通大学);Music Tech Lab, DynamiX(音乐科技实验室,DynamiX);Shanghai AI Laboratory(上海人工智能实验室);Beijing Film Academy(北京电影学院);Tsinghua University(清华大学);McGill University(麦吉尔大学);The Chinese University of Hong Kong(香港中文大学)
A Survey on Cross-Modal Interaction Between Music and Multimodal Data
MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China. E-mail: S. Li;School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100190, China. E-mail: F. Shen;Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China. E-mail: .
A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Innovation Center of Yangtze River Delta, Zhejiang University(浙江大学长三角创新中心)
BeatFM: Improving Beat Tracking with Pre-trained Music Foundation Model
School of Computer Science, Fudan University, Shanghai, China(复旦大学计算机科学学院);Naval Medical Center, PLA, China(中国人民解放军海军医疗中心);Graduate School of Informatics, Kyoto University, Kyoto, Japan(京都大学信息科学研究生院);Graduate School of Advanced Science and Engineering, Hiroshima University, Hiroshima, Japan(广岛大学先进科学与工程研究生院);Shanghai Key Laboratory of Intelligent Information Processing, Fudan University, Shanghai, China(上海智能信息处理重点实验室)