arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.14340cs.SDcs.AIcs.CLcs.LGeess.AS
收藏

音乐基础模型:综述

Foundation Models for Music: A Survey

Centre for Digital Music, Queen Mary University of London(伦敦大学玛丽皇后学院数字音乐中心);School of Computer Science, Carnegie Mellon University(卡内基梅隆大学计算机学院);University of Sheffield(谢菲尔德大学);The University of Manchester(曼彻斯特大学);University of Auckland(奥克兰大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);New York University Shanghai(纽约大学上海分校);University of Tokyo(东京大学)
Yinghao Ma, Anders Øland, Anton Ragni, Bleiz MacSen Del Sette, Charalampos Saitis, Chris Donahue, Chenghua Lin, Christos Plachouras, Emmanouil Benetos, Elona Sh… 展开作者

Yinghao Ma, Anders Øland, Anton Ragni, Bleiz MacSen Del Sette, Charalampos Saitis, Chris Donahue, Chenghua Lin, Christos Plachouras, Emmanouil Benetos, Elona Shatri, Fabio Morreale, Ge Zhang, György Fazekas, Gus Xia, Huan Zhang, Ilaria Manco, Jiawen Huang, Julien Guinot, Liwei Lin, Luca Marinelli, Max W. Y. Lam, Megha Sharma, Qiuqiang Kong, Roger B. Dannenberg, Ruibin Yuan, Shangda Wu, Shih-Lun Wu, Shuqi Dai, Shun Lei, Shiyin Kang, Simon Dixon, Wenhu Chen, Wenhao Huang, Xingjian Du, Xingwei Qu, Xu Tan, Yizhi Li, Zeyue Tian, Zhiyong Wu, Zhizheng Wu, Ziyang Ma, Ziyu Wang

AI总结:

本文系统综述了音乐领域基础模型的预训练、生成与多模态学习进展,指出表示探索不足与通用性欠缺,并强调指令微调、音乐智能体、数据集评估及伦理等未来方向。

摘要:

近年来,诸如大语言模型(LLMs)和潜在扩散模型(LDMs)等基础模型(FMs)已深刻影响包括音乐在内的多个领域。本综述系统考察了音乐领域中最先进的(SOTA)预训练模型与基础模型,涵盖表示学习、生成学习和多模态学习。我们首先阐述音乐在各行业中的重要性,并追溯人工智能在音乐中的演进历程。通过界定基础模型所针对的模态,我们发现许多音乐表示在基础模型开发中尚未得到充分探索。随后,重点讨论了以往方法在多样化音乐应用上缺乏通用性的问题,以及基础模型在音乐理解、生成和医学应用中的潜力。通过全面探讨模型预训练范式、架构选择、分词、微调方法及可控性等细节,我们强调了本应得到深入探索的重要主题,例如指令微调与上下文学习、缩放定律与涌现能力,以及长序列建模等。专门章节介绍了音乐智能体的相关见解,并深入分析了预训练和下游任务所必需的数据集与评估方法。最后,通过强调伦理考量的至关重要性,我们主张未来关于音乐基础模型的研究应更加关注可解释性、透明度、人类责任和版权等问题。本文对音乐基础模型的未来挑战与趋势提出了见解,旨在塑造音乐领域中人类与人工智能协作的发展方向。

英文摘要:

In recent years, foundation models (FMs) such as large language models (LLMs) and latent diffusion models (LDMs) have profoundly impacted diverse sectors, including music. This comprehensive review examines state-of-the-art (SOTA) pre-trained models and foundation models in music, spanning from representation learning, generative learning and multimodal learning. We first contextualise the significance of music in various industries and trace the evolution of AI in music. By delineating the modalities targeted by foundation models, we discover many of the music representations are underexplored in FM development. Then, emphasis is placed on the lack of versatility of previous methods on diverse music applications, along with the potential of FMs in music understanding, generation and medical application. By comprehensively exploring the details of the model pre-training paradigm, architectural choices, tokenisation, finetuning methodologies and controllability, we emphasise the important topics that should have been well explored, like instruction tuning and in-context learning, scaling law and emergent ability, as well as long-sequence modelling etc. A dedicated section presents insights into music agents, accompanied by a thorough analysis of datasets and evaluations essential for pre-training and downstream tasks. Finally, by underscoring the vital importance of ethical considerations, we advocate that following research on FM for music should focus more on such issues as interpretability, transparency, human responsibility, and copyright issues. The paper offers insights into future challenges and trends on FMs for music, aiming to shape the trajectory of human-AI collaboration in the music realm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑