arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.01591eess.AScs.AIcs.CLcs.SD
收藏

BAT:基于大语言模型的空间声音推理学习

BAT: Learning to Reason about Spatial Sounds with Large Language Models

Shanghai Jiao Tong University(上海交通大学)

Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, David Harwath

AI总结:

该研究提出结合Spatial-AST与LLaMA-2 7B的BAT模型,通过合成数据集训练,实现了超越SELD任务的空间声音感知与推理,展现了LLM在空间音频领域的应用潜力。

摘要:

空间声音推理是人类的一项基础技能,使我们能够基于声音来导航和解读周围环境。在本文中,我们提出了BAT,它将双耳声学场景分析模型的空间声音感知能力与大语言模型(LLM)的自然语言推理能力相结合,以复现这种先天能力。针对现有野外空间声音数据集的缺失问题,我们利用AudioSet和SoundSpaces 2.0合成了一个双耳音频数据集。接下来,我们开发了SpatialSoundQA,这是一个基于空间声音的问答数据集,提供了一系列问答任务,用于在空间声音感知和推理的各个方面训练BAT。BAT的声学前端编码器是一个名为空间音频频谱图Transformer(Spatial Audio Spectrogram Transformer,即Spatial-AST)的新型空间音频编码器,其自身在声音事件检测、空间定位和距离估计方面表现出强劲性能。通过将Spatial-AST与LLaMA-2 7B模型相结合,BAT超越了标准的声音事件定位与检测(SELD)任务,使模型能够推理环境中声音之间的关系。我们的实验表明,BAT在空间声音感知和推理方面均表现出卓越性能,彰显了LLM在导航和解读复杂空间音频环境方面的巨大潜力。

英文摘要:

Spatial sound reasoning is a fundamental human skill, enabling us to navigate and interpret our surroundings based on sound. In this paper we present BAT, which combines the spatial sound perception ability of a binaural acoustic scene analysis model with the natural language reasoning capabilities of a large language model (LLM) to replicate this innate ability. To address the lack of existing datasets of in-the-wild spatial sounds, we synthesized a binaural audio dataset using AudioSet and SoundSpaces 2.0. Next, we developed SpatialSoundQA, a spatial sound-based question-answering dataset, offering a range of QA tasks that train BAT in various aspects of spatial sound perception and reasoning. The acoustic front end encoder of BAT is a novel spatial audio encoder named Spatial Audio Spectrogram Transformer, or Spatial-AST, which by itself achieves strong performance across sound event detection, spatial localization, and distance estimation. By integrating Spatial-AST with LLaMA-2 7B model, BAT transcends standard Sound Event Localization and Detection (SELD) tasks, enabling the model to reason about the relationships between the sounds in its environment. Our experiments demonstrate BAT's superior performance on both spatial sound perception and reasoning, showcasing the immense potential of LLMs in navigating and interpreting complex spatial audio environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑