JAEGER：模拟物理环境中的联合3D音频-视觉定位与推理
JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments
AI总结：
提出JAEGER框架，通过集成RGB-D观测和多通道一阶环境声学，将音频-视觉大语言模型扩展到3D空间，实现联合空间定位与推理，并引入神经强度向量（Neural IV）提升声源方向估计的鲁棒性。
摘要：
当前的音频-视觉大语言模型（AV-LLMs）主要局限于2D感知，依赖于RGB视频和单声道音频。这种设计选择引入了基本的维度不匹配，阻碍了在复杂3D环境中可靠的声源定位和空间推理。我们通过提出JAEGER框架来解决这一限制，该框架将AV-LLMs扩展到3D空间，通过集成RGB-D观测和多通道一阶环境声学实现联合空间定位与推理。我们工作的核心贡献是神经强度向量（Neural IV），一种学习的空间音频表示，它编码了鲁棒的方向线索，以增强到达方向估计，即使在具有重叠声源的不利声学场景中也是如此。为了促进大规模训练和系统评估，我们提出了SpatialSceneQA，一个包含从模拟物理环境中整理的6.1万个指令调优样本的基准。大量实验表明，我们的方法在各种空间感知和推理任务中始终优于以2D为中心的基线，强调了显式3D建模对于推进物理环境中AI的必要性。我们的源代码、预训练模型检查点和数据集可在https://github.com/liuzhan22/JAEGER获取。
英文摘要：
Current audio-visual large language models (AV-LLMs) are predominantly restricted to 2D perception, relying on RGB video and monaural audio. This design choice introduces a fundamental dimensionality mismatch that precludes reliable source localization and spatial reasoning in complex 3D environments. We address this limitation by presenting JAEGER, a framework that extends AV-LLMs to 3D space, to enable joint spatial grounding and reasoning through the integration of RGB-D observations and multi-channel first-order ambisonics. A core contribution of our work is the neural intensity vector (Neural IV), a learned spatial audio representation that encodes robust directional cues to enhance direction-of-arrival estimation, even in adverse acoustic scenarios with overlapping sources. To facilitate large-scale training and systematic evaluation, we propose SpatialSceneQA, a benchmark of 61k instruction-tuning samples curated from simulated physical environments. Extensive experiments demonstrate that our approach consistently surpasses 2D-centric baselines across diverse spatial perception and reasoning tasks, underscoring the necessity of explicit 3D modelling for advancing AI in physical environments. Our source code, pre-trained model checkpoints, and datasets are available at https://github.com/liuzhan22/JAEGER.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
