在任意地点聆听任意声音
Hearing Anything Anywhere
AI总结：
本文提出DiffRIR可微RIR渲染框架，仅用约12个稀疏RIR录音和场景平面重建即可恢复任意环境的空间声学特性，并在真实数据上优于现有基线。
摘要：
近年来，3D计算机视觉和计算机图形学取得了巨大进展，涌现出的工具能够将真实世界的3D环境虚拟化，以服务于众多混合现实（XR）应用。然而，除了沉浸式视觉体验之外，沉浸式听觉体验对于我们对环境的整体感知同样至关重要。在本文中，我们的目标是：仅借助稀疏的一组（约12个）房间脉冲响应（RIR）录音以及场景的平面重建结果，重构任意环境的空间声学特征；这一设置很容易由普通用户实现。为此，我们提出了DiffRIR，这是一个可微的RIR渲染框架，对场景中显著声学特征采用可解释的参数化模型，包括声源指向性和表面反射率。这使我们能够利用任意源音频，在整个空间中合成新颖的听觉体验。为评估我们的方法，我们在四个多样化的真实环境中采集了一个包含RIR录音和音乐的数据集。结果表明，在未见位置渲染单声道和双耳RIR及音乐方面，我们的模型优于最先进的基线方法，并且能够学习到表征场景中声源与表面声学属性的物理可解释参数。
英文摘要：
Recent years have seen immense progress in 3D computer vision and computer graphics, with emerging tools that can virtualize real-world 3D environments for numerous Mixed Reality (XR) applications. However, alongside immersive visual experiences, immersive auditory experiences are equally vital to our holistic perception of an environment. In this paper, we aim to reconstruct the spatial acoustic characteristics of an arbitrary environment given only a sparse set of (roughly 12) room impulse response (RIR) recordings and a planar reconstruction of the scene, a setup that is easily achievable by ordinary users. To this end, we introduce DiffRIR, a differentiable RIR rendering framework with interpretable parametric models of salient acoustic features of the scene, including sound source directivity and surface reflectivity. This allows us to synthesize novel auditory experiences through the space with any source audio. To evaluate our method, we collect a dataset of RIR recordings and music in four diverse, real environments. We show that our model outperforms state-ofthe-art baselines on rendering monaural and binaural RIRs and music at unseen locations, and learns physically interpretable parameters characterizing acoustic properties of the sound source and surfaces in the scene.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
