MuChin：用于评估音乐领域语言模型的中文口语化描述基准
MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Music
AI总结：
本文提出中文口语化音乐描述基准 MuChin，通过多人多阶段标注构建高精度测试集，并用于分析人群差异、验证微调效果及评估现有音乐理解模型。
摘要：
快速发展的多模态大语言模型（LLM）迫切需要新的基准，以统一评估其在理解音乐并进行文本描述方面的性能。然而，由于音乐信息检索（MIR）算法与人类理解之间存在语义鸿沟、专业人士与公众之间存在差异，以及标注精度较低，现有音乐描述数据集无法充当基准。为此，我们提出了 MuChin，这是首个采用中文口语化语言的开源音乐描述基准，旨在评估多模态 LLM 在理解和描述音乐方面的性能。我们建立了 Caichong Music Annotation Platform（CaiMAP，菜虫音乐标注平台），该平台采用创新的多人、多阶段保障方法，并招募业余爱好者和专业人员，以确保标注精度以及与大众语义的一致性。利用该方法，我们构建了具有多维、高精度音乐标注的数据集 Caichong Music Dataset（CaiMD，菜虫音乐数据集），并精心选取 1,000 条高质量条目作为 MuChin 的测试集。基于 MuChin，我们分析了专业人士和业余爱好者在音乐描述方面的差异，并通过实验证明了标注数据对微调 LLM 的有效性。最终，我们利用 MuChin 评估了现有音乐理解模型提供口语化音乐描述的能力。与该基准相关的所有数据、评分代码和详细附录均已开源（https://github.com/CarlWangChina/MuChin/）。
英文摘要：
The rapidly evolving multimodal Large Language Models (LLMs) urgently require new benchmarks to uniformly evaluate their performance on understanding and textually describing music. However, due to semantic gaps between Music Information Retrieval (MIR) algorithms and human understanding, discrepancies between professionals and the public, and low precision of annotations, existing music description datasets cannot serve as benchmarks. To this end, we present MuChin, the first open-source music description benchmark in Chinese colloquial language, designed to evaluate the performance of multimodal LLMs in understanding and describing music. We established the Caichong Music Annotation Platform (CaiMAP) that employs an innovative multi-person, multi-stage assurance method, and recruited both amateurs and professionals to ensure the precision of annotations and alignment with popular semantics. Utilizing this method, we built a dataset with multi-dimensional, high-precision music annotations, the Caichong Music Dataset (CaiMD), and carefully selected 1,000 high-quality entries to serve as the test set for MuChin. Based on MuChin, we analyzed the discrepancies between professionals and amateurs in terms of music description, and empirically demonstrated the effectiveness of annotated data for fine-tuning LLMs. Ultimately, we employed MuChin to evaluate existing music understanding models on their ability to provide colloquial descriptions of music. All data related to the benchmark, along with the scoring code and detailed appendices, have been open-sourced (https://github.com/CarlWangChina/MuChin/).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
