Music represents a cornerstone of human culture, existing digitally across diverse modalities, including audio, symbolic encodings (e.g., MIDI, MusicXML), and sheet music. Despite the advancement of Multimodal Large Language Models (MLLMs), current music benchmarks face three major limitations. First, large static benchmarks are resource-intensive to evaluate, and it remains unclear how their results transfer to diverse kinds of music beyond those included in the benchmark. Second, benchmarks claiming to measure "music understanding" often fail to require music perception. Third, they do not support systematic performance comparisons across musical modalities. To overcome these issues, we introduce the Music I Care About Meta-Benchmark (MusICA-MetaBench), a framework that automatically derives on-demand benchmarks directly from user-provided data. By leveraging structured symbolic representations (e.g., MusicXML) and our pre-defined question templates, we build multiple-choice question-answer pairs that probe music perception competencies, aligned with music pedagogy, across audio, music notation images, and symbolic files. We demonstrate our framework with the ChoraleBricks dataset, and experimentally determine benchmark sizes that ensure statistically reliable model comparisons for this setup. By comparing against text-only and white-noise baselines, we show our questions do measure music perception. Ultimately, MusICA-MetaBench represents a significant advancement in the cross-modal assessment of music perception for MLLMs. By proposing a dataset-specific benchmarking paradigm, it enables efficient on-demand evaluation of music perception capabilities.
MSRBench: A Benchmarking Dataset for Music Source Restoration
Independent Researcher(独立研究者);John Hopkins University(约翰·霍普金斯大学);The Chinese University of Hong Kong(香港中文大学);Sony AI(索尼人工智能);King's College London(伦敦国王学院)
Benchmarking Sub-Genre Classification For Mainstage Dance Music
Johns Hopkins University(约翰霍普金斯大学);Southeast University(东南大学);National University of Defense Technology(国防科技大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Jamendo-QA: A Large-Scale Music Question Answering Dataset
Department of Artificial Intelligence, Yonsei University(延世大学人工智能系);MAAP LAB, MODULABS(MODULABS MAAP实验室);KRAFTON;AI Matics;Department of Media Software, Sungkyul University(顺天大学媒体软件系)
ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Following
Graduate School of Informatics, Kyoto University, Japan(京都大学信息科学研究生院);College of Computer Science and Artificial Intelligence, Fudan University, China(复旦大学计算机科学与人工智能学院);Graduate School of Engineering, Kyoto University, Japan(京都大学工学研究科)
MERGE -- A Bimodal Audio-Lyrics Dataset for Static Music Emotion Recognition
University of Coimbra(科英布拉大学);Centre for Informatics and Systems of the University of Coimbra(科英布拉大学信息与系统中心);Department of Informatics Engineering(信息工程系);LASI;Polytechnic Institute of Leiria - School of Technology and Management(莱里亚理工学院-技术与管理学院);Ci2 - Smart Cities Research Center(Ci2-智慧城市研究中心)
Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio
University of California, San Diego, Computer Science and Engineering Department(加州大学圣地亚哥分校计算机科学与工程系);Carnegie Mellon University, School of Computer Science(卡内基梅隆大学计算机科学学院)
UniVerse: Benchmarking and Enhancing LALMs on Culturally Inclusive Low-Resource Music Understanding
Sogang University(西江大学);KAIST(韩国科学技术院);NYU Shanghai(上海纽约大学);JIUTIAN Research, China Mobile(中国移动九天研究院);The State Key Laboratory of Multimedia Information Processing, Peking University(北京大学多媒体信息处理国家重点实验室);China Mobile (Hong Kong) Innovation Research Institute(中国移动(香港)创新研究院);Central Conservatory of Music(中央音乐学院)
Benchmarking Early Agitation Prediction in Community-Dwelling People with Dementia Using Multimodal Sensors and Machine Learning
KITE Research Institute(KITE研究机构);Toronto Rehabilitation Institute(多伦多康复研究所);University Health Network(大学健康网络);Lawrence Bloomberg Faculty of Nursing(劳伦斯布罗德利护理学院);University of Toronto(多伦多大学);College of Engineering and Technology(工程与技术学院);American University of the Middle East(中东美国大学)
A Comprehensive Benchmark for COVID-19 Predictive Modeling Using Electronic Health Records in Intensive Care
University of Edinburgh(爱丁堡大学);Health Data Research UK(英国健康数据研究中心);Peking University(北京大学);Peking University Third Hospital(北京大学第三医院);Kaihong Digital(凯宏数字)
LMU Munich(慕尼黑大学);University of Freiburg(弗莱堡大学);Eindhoven University of Technology(埃因霍温理工大学);Bosch Center for Artificial Intelligence(博世人工智能中心);TU Dortmund University(多特蒙德工业大学);Federal Technology University Paraná (UTFPR)(巴拉那联邦科技大学);Leiden University(莱顿大学)
Benchmarking Machine Learning Architectures for Antimicrobial Stewardship in Pediatric ICUs
Department of Intensive Care and Neonatology and Children’s Research Center University of Zurich University Children’s Hospital Zurich(重症护理与新生儿科及儿童研究中心,苏黎世大学苏黎世儿童医院);Department of Health Sciences and Technology ETH Zurich(健康科学与技术系,苏黎世联邦理工学院);Department of Computer Science ETH Zurich(计算机科学系,苏黎世联邦理工学院)
Benchmarking emergency department triage prediction models with machine learning and large public electronic health records
Duke-NUS Medical School(杜克-新加坡国立大学医学院);Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局高性能计算研究所);Nanyang Technological University(南洋理工大学);Monash University(莫纳什大学);National University of Singapore(新加坡国立大学);Duke University(杜克大学);Beth Israel Deaconess Medical Center(贝斯以色列女执事医疗中心);Harvard Medical School(哈佛医学院);Massachusetts Institute of Technology(麻省理工学院);Singapore General Hospital(新加坡中央医院);Singapore Health Services(新加坡保健服务集团)
CrossMuSim: A Cross-Modal Framework for Music Similarity Retrieval with LLM-Powered Text Description Sourcing and Mining
Audio Lab Hong Kong, Hong Kong Research Center, Huawei Technologies, Hong Kong SAR(香港音频实验室、香港研究中心、华为技术有限公司、香港特别行政区);Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain(音乐技术组、庞培法华大学、巴塞罗那西班牙);Munich Research Center, Huawei Technologies, Munich, Germany(慕尼黑研究中心、华为技术有限公司、慕尼黑德国)
YMIR: A new Benchmark Dataset and Model for Arabic Yemeni Music Genre Classification Using Convolutional Neural Networks
School of Artificial Intelligence, Xidian University(西安电子科技大学人工智能学院);Department of Information Science and Technology, Northwest University(西北大学信息科学与技术学院)
MixFake: Benchmarking and Enhancing Audio Deepfake Detection in Diverse Real-world Mixed Audio
School of Cyber Science and Engineering, Nanjing University of Science and Technology, Nanjing, China(南京理工大学信息科学与工程学院);The State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China(浙江大学区块链与数据安全国家重点实验室);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, Hangzhou, China(杭州高新技术区(滨江)区块链与数据安全研究院)