Music Understanding LLaMA：通过问答与字幕生成推进文本到音乐生成
Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning
AI总结：
针对文本到音乐生成缺乏带字幕的大规模公开数据集的问题，提出能完成音乐问答与字幕生成的MU-LLaMA模型，配套构建MusicQA数据集，实验显示其性能超越现有SOTA模型。
摘要：
文本到音乐生成（T2M-Gen）面临的一大障碍是缺乏带自然语言字幕的大规模公开音乐数据集。为解决该问题，我们提出Music Understanding LLaMA（MU-LLaMA，音乐理解大语言模型），它能够回答音乐相关问题并为音乐文件生成字幕。我们的模型利用预训练MERT模型的音频表示来提取音乐特征。然而，获取适合训练MU-LLaMA的数据集仍具挑战，因为现有公开音频问答数据集缺乏开放式音乐问答所需的深度。为填补这一空白，我们提出一种从现有音频字幕数据集生成问答对的方法，并推出专为开放式音乐相关问答设计的MusicQA Dataset。实验表明，在我们设计的MusicQA Dataset上训练的MU-LLaMA模型，在音乐问答和音乐字幕生成两项任务的各类指标上均取得优异表现，超越了这两个领域当前的最先进（SOTA）模型，为T2M-Gen研究领域带来了颇具前景的进展。
英文摘要：
Text-to-music generation (T2M-Gen) faces a major obstacle due to the scarcity of large-scale publicly available music datasets with natural language captions. To address this, we propose the Music Understanding LLaMA (MU-LLaMA), capable of answering music-related questions and generating captions for music files. Our model utilizes audio representations from a pretrained MERT model to extract music features. However, obtaining a suitable dataset for training the MU-LLaMA model remains challenging, as existing publicly accessible audio question answering datasets lack the necessary depth for open-ended music question answering. To fill this gap, we present a methodology for generating question-answer pairs from existing audio captioning datasets and introduce the MusicQA Dataset designed for answering open-ended music-related questions. The experiments demonstrate that the proposed MU-LLaMA model, trained on our designed MusicQA dataset, achieves outstanding performance in both music question answering and music caption generation across various metrics, outperforming current state-of-the-art (SOTA) models in both fields and offering a promising advancement in the T2M-Gen research field.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
