Listen2Scene：面向重建三维场景的交互式材质感知双耳声音传播
Listen2Scene: Interactive material-aware binaural sound propagation for reconstructed 3D scenes
AI总结：
Listen2Scene提出一种基于图神经网络和条件生成对抗网络的端到端双耳声音传播方法，利用三维场景材质与拓扑生成声学效果，在RTX 2080 Ti上0.1毫秒内完成，经感知和定量评估优于现有方法。
摘要：
我们提出了一种用于虚拟现实（VR）和增强现实（AR）应用的端到端双耳音频渲染方法（Listen2Scene）。我们提出了一种新颖的基于神经网络的双耳声音传播方法，用于为真实环境的室内三维模型生成声学效果。任何干净音频或干音频都可以与生成的声学效果进行卷积，以渲染对应于真实环境的音频。我们提出了一种图神经网络，它利用三维场景的材质和拓扑信息，生成场景潜在向量。此外，我们使用条件生成对抗网络（CGAN）从场景潜在向量生成声学效果。我们的网络可以处理重建三维网格模型中的孔洞或其他伪影。我们为生成器网络提出了一种高效的代价函数，以融入空间音频效果。给定声源和听者位置，我们的基于学习的双耳声音传播方法可以在NVIDIA GeForce RTX 2080 Ti GPU上于0.1毫秒内生成声学效果。我们使用交互式几何声音传播算法生成的双耳声学效果以及捕获的真实声学效果/真实世界录音来评估我们方法的准确性。我们还进行了感知评估，观察到我们方法渲染的音频比使用先前基于学习和基于几何的声音传播算法渲染的音频更逼真。我们使用统计声学参数和能量衰减曲线定量评估了我们方法的准确性。演示视频、代码和数据集可在线获取（https://anton-jeran.github.io/Listen2Scene/）。
英文摘要：
We present an end-to-end binaural audio rendering approach (Listen2Scene) for virtual reality (VR) and augmented reality (AR) applications. We propose a novel neural-network-based binaural sound propagation method to generate acoustic effects for indoor 3D models of real environments. Any clean audio or dry audio can be convolved with the generated acoustic effects to render audio corresponding to the real environment. We propose a graph neural network that uses both the material and the topology information of the 3D scenes and generates a scene latent vector. Moreover, we use a conditional generative adversarial network (CGAN) to generate acoustic effects from the scene latent vector. Our network can handle holes or other artifacts in the reconstructed 3D mesh model. We present an efficient cost function for the generator network to incorporate spatial audio effects. Given the source and the listener position, our learning-based binaural sound propagation approach can generate an acoustic effect in 0.1 milliseconds on an NVIDIA GeForce RTX 2080 Ti GPU. We have evaluated the accuracy of our approach with binaural acoustic effects generated using an interactive geometric sound propagation algorithm and captured real acoustic effects / real-world recordings. We also performed a perceptual evaluation and observed that the audio rendered by our approach is more plausible than audio rendered using prior learning-based and geometric-based sound propagation algorithms. We quantitatively evaluated the accuracy of our approach using statistical acoustic parameters, and energy decay curves. The demo videos, code and dataset are available online (https://anton-jeran.github.io/Listen2Scene/).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
