arXiv 2505.130322025-05-19cs.SDcs.CLcs.MMeess.AS

MMAR：面向语音、音频、音乐及其混合场景深度推理的挑战性基准

MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix

Shanghai Jiao Tong University（上海交通大学）；Nanyang Technological University（南洋理工大学）；Queen Mary University of London（伦敦玛丽女王大学）；ByteDance（字节跳动）；Shanghai Innovation Institute（上海创新研究院）；Tsinghua University（清华大学）；University of Chinese Academy of Sciences（中国科学院大学）；The Hong Kong University of Science and Technology（香港科学与技术大学）；The University of Texas at Austin（得克萨斯大学奥斯汀分校）

Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, Kai Li, Keliang Li, Siyou Li, Xinfeng Li… 展开作者

Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, Kai Li, Keliang Li, Siyou Li, Xinfeng Li, Xiquan Li, Zheng Lian, Yuzhe Liang, Minghao Liu, Zhikang Niu, Tianrui Wang, Yuping Wang, Yuxuan Wang, Yihao Wu, Guanrou Yang, Jianwei Yu, Ruibin Yuan, Zhisheng Zheng, Ziya Zhou, Haina Zhu, Wei Xue, Emmanouil Benetos, Kai Yu, Eng-Siong Chng, Xie Chen

备注：: Open-source at https://github.com/ddlBoJack/MMAR

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究推出覆盖语音、音频、音乐及混合场景的多学科深度推理基准MMAR，含1000个分层标注的三元组与思维链依据，评估发现当前各类模型存在明显能力局限，可为音频推理领域研究提供支撑。

摘要：

我们推出MMAR，这是一个全新基准，旨在评估音频-语言模型（Audio-Language Models, ALMs）在海量多学科任务中的深度推理能力。MMAR包含1000个精心筛选的音频-问题-答案三元组，均采集自真实互联网视频，并经过多轮错误修正与质量检查以确保高水准。\n现有基准多局限于声音、音乐或语音的特定领域，MMAR则突破这一限制，覆盖更广泛的真实音频场景，包括声音、音乐与语音的多模态混合组合。MMAR中的每个问题都按四个推理层级进行分层分类：信号层、感知层、语义层与文化层，每个层级下还设有子类别，以体现任务的多样性与复杂性。\n为进一步推动该领域研究，我们为每个问题标注了思维链（Chain-of-Thought, CoT）推理依据，助力未来音频推理技术的发展。该基准中的每个题目都要求超越表层理解的多步深度推理，此外还有部分问题需要研究生水平的感知能力与特定领域知识，进一步提升了基准的难度与深度。\n我们采用音频字幕输入方式，使用多种模型对MMAR进行了评估，涵盖大型音频-语言模型（Large Audio-Language Models, LALMs）、大型音频推理模型（Large Audio Reasoning Models, LARMs）、全模态语言模型（Omni Language Models, OLMs）、大型语言模型（Large Language Models, LLMs）以及大型推理模型（Large Reasoning Models, LRMs）。这些模型在MMAR上的表现凸显了该基准的挑战性，我们的分析也进一步揭示了当前模型在理解与推理能力上的关键局限。我们希望MMAR能够成为推动这一重要但探索不足领域未来发展的催化剂。

英文摘要：

We introduce MMAR, a new benchmark designed to evaluate the deep reasoning capabilities of Audio-Language Models (ALMs) across massive multi-disciplinary tasks. MMAR comprises 1,000 meticulously curated audio-question-answer triplets, collected from real-world internet videos and refined through iterative error corrections and quality checks to ensure high quality. Unlike existing benchmarks that are limited to specific domains of sound, music, or speech, MMAR extends them to a broad spectrum of real-world audio scenarios, including mixed-modality combinations of sound, music, and speech. Each question in MMAR is hierarchically categorized across four reasoning layers: Signal, Perception, Semantic, and Cultural, with additional sub-categories within each layer to reflect task diversity and complexity. To further foster research in this area, we annotate every question with a Chain-of-Thought (CoT) rationale to promote future advancements in audio reasoning. Each item in the benchmark demands multi-step deep reasoning beyond surface-level understanding. Moreover, a part of the questions requires graduate-level perceptual and domain-specific knowledge, elevating the benchmark's difficulty and depth. We evaluate MMAR using a broad set of models, including Large Audio-Language Models (LALMs), Large Audio Reasoning Models (LARMs), Omni Language Models (OLMs), Large Language Models (LLMs), and Large Reasoning Models (LRMs), with audio caption inputs. The performance of these models on MMAR highlights the benchmark's challenging nature, and our analysis further reveals critical limitations of understanding and reasoning capabilities among current models. We hope MMAR will serve as a catalyst for future advances in this important but little-explored area.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。