arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.01153cs.CL
收藏

SongSage:一种具有歌词生成预训练的大型音乐语言模型

SongSage: A Large Musical Language Model with Lyric Generative Pre-training

School of Computer Science, Wuhan University(武汉大学计算机学院);School of Information Management, Wuhan University(武汉大学信息管理学院);Tsinghua University(清华大学);School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院)

Jiani Guo, Jiajia Li, Jie Wu, Zuchao Li, Yujiu Yang, Ping Wang

AI总结:

SongSage是一种通过歌词生成预训练的大型音乐语言模型,专为提升歌词理解和播放列表推荐能力而设计,同时具备一般知识理解能力。

摘要:

大型语言模型在各种领域取得了显著成功,但其对以歌词为中心的知识的理解尚未得到充分探索。在本工作中,我们首先引入了PlaylistSense数据集,用于评估语言模型对播放列表的理解能力。PlaylistSense包含十种源自常见现实视角的用户查询,挑战LLMs准确把握播放列表特征并处理多样的用户意图。全面评估表明,当前通用语言模型在播放列表理解方面仍有提升空间。受此启发,我们引入了SongSage,一种配备多样化歌词中心智能的大型音乐语言模型,通过歌词生成预训练进行训练。SongSage在LyricBank上进行持续预训练,LyricBank是一个精心编纂的包含54.8亿个标记的歌词内容语料库,随后通过LyricBank-SFT进行微调,LyricBank-SFT是一个包含775,000个样本的精心设计的指令集,涵盖九个核心歌词中心任务。实验结果表明,SongSage对歌词中心知识有很强的理解能力,能够有效重写用户查询以实现零样本播放列表推荐,生成并继续歌词效果良好,并在七个其他能力上表现娴熟。除了其歌词中心的专业能力外,SongSage还保留了一般知识的理解能力,并在MMLU评分上取得了竞争性的成绩。由于版权限制,我们将数据集保持不可访问,并发布SongSage和训练脚本以确保可重复性和支持音乐AI研究和应用,数据集发布计划的细节在附录中提供。

英文摘要:

Large language models have achieved significant success in various domains, yet their understanding of lyric-centric knowledge has not been fully explored. In this work, we first introduce PlaylistSense, a dataset to evaluate the playlist understanding capability of language models. PlaylistSense encompasses ten types of user queries derived from common real-world perspectives, challenging LLMs to accurately grasp playlist features and address diverse user intents. Comprehensive evaluations indicate that current general-purpose LLMs still have potential for improvement in playlist understanding. Inspired by this, we introduce SongSage, a large musical language model equipped with diverse lyric-centric intelligence through lyric generative pretraining. SongSage undergoes continual pretraining on LyricBank, a carefully curated corpus of 5.48 billion tokens focused on lyrical content, followed by fine-tuning with LyricBank-SFT, a meticulously crafted instruction set comprising 775k samples across nine core lyric-centric tasks. Experimental results demonstrate that SongSage exhibits a strong understanding of lyric-centric knowledge, excels in rewriting user queries for zero-shot playlist recommendations, generates and continues lyrics effectively, and performs proficiently across seven additional capabilities. Beyond its lyric-centric expertise, SongSage also retains general knowledge comprehension and achieves a competitive MMLU score. We will keep the datasets inaccessible due to copyright restrictions and release the SongSage and training script to ensure reproducibility and support music AI research and applications, the datasets release plan details are provided in the appendix.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑