FoleySpace：视觉对齐的双耳空间音频生成
FoleySpace: Vision-Aligned Binaural Spatial Audio Generation
AI总结：
FoleySpace提出一种视频到双耳音频生成框架，通过声源二维坐标与深度估计、三维轨迹映射并结合预训练V2A单声道音频作为扩散模型条件，实现视觉对齐的空间一致立体声生成，并在空间感知一致性上优于现有方法。
摘要：
近年来，随着AIGC的发展，基于深度学习的视频到音频（V2A）技术引起了广泛关注。然而，现有研究大多集中于缺乏空间感知的单声道音频生成，而对能够提供更强沉浸感的双耳空间音频生成技术的探索仍然不足。为解决这一问题，我们提出了FoleySpace，一个用于视频到双耳音频生成的框架，该框架在视觉信息的引导下生成沉浸式且空间一致的立体声。具体来说，我们开发了一种声源估计方法，用于确定每个视频帧中声源的二维坐标和深度，然后采用坐标映射机制将二维声源位置转换为三维轨迹。该三维轨迹与预训练V2A模型生成的单声道音频一起，作为扩散模型的条件输入，以生成空间一致的双耳音频。为支持动态声场生成，我们基于录制的头相关脉冲响应构建了一个训练数据集，其中包含多种声源运动场景。实验结果表明，所提方法在空间感知一致性方面优于现有方法，有效增强了视听体验的沉浸感。
英文摘要：
Recently, with the advancement of AIGC, deep learning-based video-to-audio (V2A) technology has garnered significant attention. However, existing research mostly focuses on mono audio generation that lacks spatial perception, while the exploration of binaural spatial audio generation technologies, which can provide a stronger sense of immersion, remains insufficient. To solve this problem, we propose FoleySpace, a framework for video-to-binaural audio generation that produces immersive and spatially consistent stereo sound guided by visual information. Specifically, we develop a sound source estimation method to determine the sound source 2D coordinates and depth in each video frame, and then employ a coordinate mapping mechanism to convert the 2D source positions into a 3D trajectory. This 3D trajectory, together with the monaural audio generated by a pre-trained V2A model, serves as a conditioning input for a diffusion model to generate spatially consistent binaural audio. To support the generation of dynamic sound fields, we constructed a training dataset based on recorded Head-Related Impulse Responses that includes various sound source movement scenarios. Experimental results demonstrate that the proposed method outperforms existing approaches in spatial perception consistency, effectively enhancing the immersive quality of the audio-visual experience.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
