跨模态生成模型用于视觉引导的双耳立体声生成
Cross-modal Generative Model for Visual-Guided Binaural Stereo Generation
AI总结：
提出视觉引导的生成对抗方法，利用共享时空视觉信息从单声道生成双耳立体声，并首次提出空间感知评估指标，在2个数据集和5个指标上达到最优性能。
摘要：
双耳立体声音频通过模仿人耳接收声音的方式进行录制，为用户提供沉浸式的听觉体验。现有方法利用自编码器并直接利用视觉空间信息来合成双耳立体声，导致视觉引导的表征能力有限。我们首次提出了一种视觉引导的生成对抗方法，用于从单声道音频生成双耳立体声音频。具体而言，我们开发了一个立体声音频生成模型（SAGM），该模型利用共享的时空视觉信息来分别引导生成器和判别器工作。共享的视觉信息在生成对抗阶段交替更新，使得生成器和判别器在视觉共享的同时传递各自的引导知识。所提出的方法学习了双向互补的视觉信息，有助于增强视觉引导在生成过程中的表达能力。此外，空间感知是双耳立体声音频的关键属性，因此对立体声空间感知的评估至关重要。然而，以往的指标未能衡量音频的空间感知能力。为此，我们首次提出了一种用于衡量音频空间感知的指标。该指标能够测量时间维度上空间感知的幅度和方向。进一步地，考虑到其功能，在一定程度上可以用它替代需要用户研究的方法。所提出的方法在2个数据集和5个评估指标上达到了最先进的性能。定性实验和用户研究表明，该方法能够生成空间逼真的立体声音频。
英文摘要：
Binaural stereo audio is recorded by imitating the way the human ear receives sound, which provides people with an immersive listening experience. Existing approaches leverage autoencoders and directly exploit visual spatial information to synthesize binaural stereo, resulting in a limited representation of visual guidance. For the first time, we propose a visually guided generative adversarial approach for generating binaural stereo audio from mono audio. Specifically, we develop a Stereo Audio Generation Model (SAGM), which utilizes shared spatio-temporal visual information to guide the generator and the discriminator to work separately. The shared visual information is updated alternately in the generative adversarial stage, allowing the generator and discriminator to deliver their respective guided knowledge while visually sharing. The proposed method learns bidirectional complementary visual information, which facilitates the expression of visual guidance in generation. In addition, spatial perception is a crucial attribute of binaural stereo audio, and thus the evaluation of stereo spatial perception is essential. However, previous metrics failed to measure the spatial perception of audio. To this end, a metric to measure the spatial perception of audio is proposed for the first time. The proposed metric is capable of measuring the magnitude and direction of spatial perception in the temporal dimension. Further, considering its function, it is feasible to utilize it instead of demanding user studies to some extent. The proposed method achieves state-of-the-art performance on 2 datasets and 5 evaluation metrics. Qualitative experiments and user studies demonstrate that the method generates space-realistic stereo audio.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
