SCENEBench:一个基于辅助和工业用例的音频理解基准测试
SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases
AI总结:
SCENEBench通过评估音频理解的四个现实场景,揭示了现有LALMs在不同任务中的性能差异,为改进模型能力提供指导。
摘要:
大规模语言模型(LLMs)的进步使音频处理能力显著提升,现在被称为大型音频语言模型(LALMs)。然而,针对超越自动语音识别(ASR)的音频理解的研究工作非常有限。本文通过提出一个基准测试套件SCENEBench(空间、跨语言、环境、非语音评估),填补这一空白。该基准测试针对四种现实世界类别中的广泛音频理解:背景声音理解、噪声定位、跨语言语音理解以及语音特征识别。这四个类别是基于辅助技术和工业噪声监控中未被充分研究的需求选择的。除了性能外,我们还测量模型延迟。该基准测试的目的在于评估音频不仅仅是所说的内容,而是如何表达以及音频的非语音成分。由于我们的音频样本是合成构建的(例如,通过叠加两个自然音频样本),我们进一步将基准测试验证于每个任务20个自然音频项目,这些项目是从现有数据集中子采样得到以匹配我们的任务标准,以评估生态效度。我们评估了五种最先进的LALMs,并发现存在关键差距:性能在不同任务中有所变化,一些任务的表现低于随机猜测,而其他任务则实现了高准确度。这些结果为模型能力的定向改进提供了方向。
英文摘要:
Advances in large language models (LLMs) have enabled significant capabilities in audio processing, resulting in state-of-the-art models now known as Large Audio Language Models (LALMs). However, minimal work has been done to measure audio understanding beyond automatic speech recognition (ASR). This paper closes that gap by proposing a benchmark suite, SCENEBench (Spatial, Cross-lingual, Environmental, Non-speech Evaluation), that targets a broad form of audio comprehension across four real-world categories: background sound understanding, noise localization, cross-linguistic speech understanding, and vocal characterizer recognition. These four categories are selected based on understudied needs from accessibility technology and industrial noise monitoring. In addition to performance, we also measure model latency. The purpose of this benchmark suite is to assess audio beyond just what words are said - rather, how they are said and the non-speech components of the audio. Because our audio samples are synthetically constructed (e.g., by overlaying two natural audio samples), we further validate our benchmark against 20 natural audio items per task, sub-sampled from existing datasets to match our task criteria, to assess ecological validity. We assess five state-of-the-art LALMs and find critical gaps: performance varies across tasks, with some tasks performing below random chance and others achieving high accuracy. These results provide direction for targeted improvements in model capabilities.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。