arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.24693cs.SDcs.CLeess.AS
收藏

STAR-Bench:探测深度时空推理作为音频4D智能

STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence

Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Shanghai Innovation Institute(上海创新研究院);Beihang University(北京航空航天大学);Shanghai Jiao Tong University(上海交通大学)

Zihan Liu, Zhikang Niu, Qiuyang Xiao, Zhisheng Zheng, Ruoqi Yuan, Yuhang Zang, Yuhang Cao, Xiaoyi Dong, Jianze Liang, Xie Chen, Leilei Sun, Dahua Lin, Jiaqi Wang

AI总结:

STAR-Bench通过测试音频4D智能,揭示了模型在细粒度感知和推理上的不足,为未来模型发展提供方向。

摘要:

尽管多模态大语言模型和大音频-语言模型在快速进步,现有音频基准测试主要测试可以从文本描述中恢复的语义,掩盖了细粒度感知推理的不足。我们正式定义音频4D智能为在时间和三维空间中对声音动态进行推理,并引入STAR-Bench来衡量它。STAR-Bench结合了基础的声学感知设置(六个属性在绝对和相对模式下)和整体时空推理设置,后者包括对连续和离散过程的片段重排序以及涵盖静态定位、多源关系和动态轨迹的空间任务。我们的数据整理流程使用两种方法确保高质量的样本。对于基础任务,我们使用程序生成和物理模拟的音频。对于整体数据,我们遵循一个四阶段过程,包括人工标注和最终选择基于人类表现。与先前基准中仅使用文本描述回答减少准确性略有不同,STAR-Bench导致更大的下降(-31.5%时间,-35.2%空间),证明其专注于语言难以描述的提示。评估19个模型揭示了与人类相比的显著差距以及能力层次:封闭源模型受限于细粒度感知,而开源模型在感知、知识和推理方面都滞后。我们的STAR-Bench为未来模型的发展提供了关键见解和清晰的发展路径,以更牢固地理解物理世界。

英文摘要:

Despite rapid progress in Multi-modal Large Language Models and Large Audio-Language Models, existing audio benchmarks largely test semantics that can be recovered from text captions, masking deficits in fine-grained perceptual reasoning. We formalize audio 4D intelligence that is defined as reasoning over sound dynamics in time and 3D space, and introduce STAR-Bench to measure it. STAR-Bench combines a Foundational Acoustic Perception setting (six attributes under absolute and relative regimes) with a Holistic Spatio-Temporal Reasoning setting that includes segment reordering for continuous and discrete processes and spatial tasks spanning static localization, multi-source relations, and dynamic trajectories. Our data curation pipeline uses two methods to ensure high-quality samples. For foundational tasks, we use procedurally synthesized and physics-simulated audio. For holistic data, we follow a four-stage process that includes human annotation and final selection based on human performance. Unlike prior benchmarks where caption-only answering reduces accuracy slightly, STAR-Bench induces far larger drops (-31.5\% temporal, -35.2\% spatial), evidencing its focus on linguistically hard-to-describe cues. Evaluating 19 models reveals substantial gaps compared with humans and a capability hierarchy: closed-source models are bottlenecked by fine-grained perception, while open-source models lag across perception, knowledge, and reasoning. Our STAR-Bench provides critical insights and a clear path forward for developing future models with a more robust understanding of the physical world.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑