无需双耳音频的可视双耳音频生成
Visually Informed Binaural Audio Generation without Binaural Audios
摘要：
立体声音频，特别是双耳音频，在沉浸式观看环境中起着至关重要的作用。最近的研究已经探索了生成由多声道音频集合监督的视觉引导的立体声音频。然而，由于专业记录设备的要求，现有的数据集在规模和种类上受到限制，这阻碍了监督方法在真实世界场景中的推广。在这项工作中，我们提出了伪双耳，一个有效的管道，是免费的双耳录音。关键的洞察力是用单声道数据仔细构建伪视觉立体声对进行训练。具体来说，我们利用球谐分解和头部相关脉冲响应（HRIR）来识别空间位置和接收到的双耳音频之间的关系。然后在视觉模态中，单声道数据的对应视觉线索被手动放置在声源位置处以形成对。与完全监督的范例相比，我们的无双耳记录管道在跨数据集评估中表现出很大的稳定性，并在主观偏好下实现了相当的性能。此外，结合双耳录音，我们的方法能够进一步提高监督设置下的双耳音频生成的性能。
英文摘要：
Stereophonic audio, especially binaural audio, plays an essential role in immersive viewing environments. Recent research has explored generating visually guided stereophonic audios supervised by multi-channel audio collections. However, due to the requirement of professional recording devices, existing datasets are limited in scale and variety, which impedes the generalization of supervised methods in real-world scenarios. In this work, we propose PseudoBinaural, an effective pipeline that is free of binaural recordings. The key insight is to carefully build pseudo visual-stereo pairs with mono data for training. Specifically, we leverage spherical harmonic decomposition and head-related impulse response (HRIR) to identify the relationship between spatial locations and received binaural audios. Then in the visual modality, corresponding visual cues of the mono data are manually placed at sound source positions to form the pairs. Compared to fully-supervised paradigms, our binaural-recording-free pipeline shows great stability in cross-dataset evaluation and achieves comparable performance under subjective preference. Moreover, combined with binaural recordings, our method is able to further boost the performance of binaural audio generation under supervised settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
