MOSA: Music Motion with Semantic Annotation Dataset for Cross-Modal Music Processing
Reid School of Music, University of Edinburgh（爱丁堡大学里德音乐学院）；Institute for Sport, Physical Education and Health Sciences (ISPEHS), University of Edinburgh（爱丁堡大学体育教育与健康科学研究所）；National Yang Ming Chiao Tung University（国立阳明交通大学）；Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所）；Graduate School of Informatics, Kyoto University（京都大学情报学研究科）
在跨模态音乐处理中，视觉、听觉和语义内容之间的转换既带来了新的可能性，也带来了挑战。构建这样一种变革性方案依赖于一个具有全面数据基础设施的基准语料库。特别是，大规模跨模态数据集的构建面临重大挑战。在本文中，我们提出了 MOSA（Music mOtion with Semantic Annotation，带语义标注的音乐动作）数据集，该数据集包含高质量的 3D 动作捕捉数据、对齐的音频录音，以及逐音符的语义标注，涵盖音高、节拍、乐句、力度、演奏法和和声，涉及 23 位专业音乐家的 742 场专业音乐表演，总计超过 30 小时和 570 K 个音符的数据。据我们所知，这是迄今为止最大的具有音符级标注的跨模态音乐数据集。为了展示 MOSA 数据集的用途，我们提出了若干创新的跨模态音乐信息检索（MIR）和音乐内容生成任务，包括从音频、视频和动作数据中检测节拍、强拍、乐句和表现性内容，以及从给定的音乐音频生成音乐家的身体动作。数据集和代码随本论文一并提供（https://github.com/yufenhuang/MOSA-Music-mOtion-and-Semantic-Annotation-dataset）。
英文摘要：
In cross-modal music processing, translation between visual, auditory, and semantic content opens up new possibilities as well as challenges. The construction of such a transformative scheme depends upon a benchmark corpus with a comprehensive data infrastructure. In particular, the assembly of a large-scale cross-modal dataset presents major challenges. In this paper, we present the MOSA (Music mOtion with Semantic Annotation) dataset, which contains high quality 3-D motion capture data, aligned audio recordings, and note-by-note semantic annotations of pitch, beat, phrase, dynamic, articulation, and harmony for 742 professional music performances by 23 professional musicians, comprising more than 30 hours and 570 K notes of data. To our knowledge, this is the largest cross-modal music dataset with note-level annotations to date. To demonstrate the usage of the MOSA dataset, we present several innovative cross-modal music information retrieval (MIR) and musical content generation tasks, including the detection of beats, downbeats, phrase, and expressive contents from audio, video and motion data, and the generation of musicians' body motion from given music audio. The dataset and codes are available alongside this publication (https://github.com/yufenhuang/MOSA-Music-mOtion-and-Semantic-Annotation-dataset).
A Survey on Cross-Modal Interaction Between Music and Multimodal Data
MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China. E-mail: S. Li；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100190, China. E-mail: F. Shen；Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China. E-mail: .
Unified Cross-modal Translation of Score Images, Symbolic Music, and Performance Audio
Department of Artificial Intelligence, Sogang University（西江大学人工智能系）；Sogang Future Lab, Sogang University（西江大学未来实验室）；Department of Physics Education, Seoul National University（首尔大学物理教育系）；Computer Science Department, Carnegie Mellon University（卡内基梅隆大学计算机科学系）；Department of Art & Technology, Sogang University（西江大学艺术与技术系）
MoSAT: Human Motion Generation from Spatial Audio and Textual Description
The University of Hong Kong（香港大学）；Massachusetts Institute of Technology（麻省理工学院）；University of Pennsylvania（宾夕法尼亚大学）；Brown University（布朗大学）；Shanghai AI Lab（上海人工智能实验室）；Macau University of Science and Technology（澳门科技大学）；Hong Kong University of Science and Technology（香港科技大学）；Texas A&M University（德克萨斯农工大学）
CrossMuSim: A Cross-Modal Framework for Music Similarity Retrieval with LLM-Powered Text Description Sourcing and Mining
Audio Lab Hong Kong, Hong Kong Research Center, Huawei Technologies, Hong Kong SAR（香港音频实验室、香港研究中心、华为技术有限公司、香港特别行政区）；Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain（音乐技术组、庞培法华大学、巴塞罗那西班牙）；Munich Research Center, Huawei Technologies, Munich, Germany（慕尼黑研究中心、华为技术有限公司、慕尼黑德国）
A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Innovation Center of Yangtze River Delta, Zhejiang University（浙江大学长三角创新中心）
MusicMamba: A Dual-Feature Modeling Approach for Generating Chinese Traditional Music with Modal Precision
School of Computer Science and Artificial Intelligence, Wuhan University of Technology（武汉理工大学计算机科学与人工智能学院）；School of Computer Science, Hubei University of Technology（湖北工业大学计算机学院）
Multi-Modality in Music: Predicting Emotion in Music from High-Level Audio Features and Lyrics
University of Copenhagen（哥本哈根大学）
本文旨在测试多模态方法的音乐情感识别（MER）是否比单模态的高层次的歌曲特征和歌词。我们使用从Spotify API检索的11个歌曲特征，组合歌词特征，包括情感，TF-IDF和Anew，以预测Deezer情绪检测数据集（DMDD）（Delbouys et al.，2018），具有4种不同的回归模型。我们发现，在11个高级歌曲特征中，主要有5个对性能有贡献，多模态特征在预测效价时比单独的音频做得更好。我们使我们的代码可供使用。
Deep Cross-Modal Correlation Learning for Audio and Lyrics in Music Retrieval
National Institute of Informatics（国立信息学研究所）；The University of Electro-Communications（电气通信大学）；Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等技术学院）；Hong Kong University of Science and Technology（香港科技大学）
Exploring Multi-Modal Control in Music-Driven Dance Generation
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Peng Cheng Laboratory（鹏城实验室）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
