CCStereo：用于双耳音频生成的音视频上下文与对比学习
CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generation
AI总结：
提出CCStereo模型，通过音视频条件归一化层和对比学习增强双耳音频生成的空间敏感性，并在FAIR-Play和MUSIC-Stereo基准上取得最先进性能。
摘要：
双耳音频生成（BAG）旨在利用视觉提示将单声道音频转换为立体声音频，这需要对空间和语义信息有深入理解。然而，当前模型存在过拟合房间环境并丢失细粒度空间细节的风险。本文提出了一种新的音视频双耳生成模型，该模型引入音视频条件归一化层，利用视觉上下文动态对齐目标差分音频特征的均值与方差，并结合一种新的对比学习方法，通过从打乱的视觉特征中挖掘负样本来增强空间敏感性。我们还引入了一种成本高效的方式，在视频数据中利用测试时增强以提升性能。我们的方法在FAIR-Play和MUSIC-Stereo基准上达到了最先进的生成精度。
英文摘要：
Binaural audio generation (BAG) aims to convert monaural audio to stereo audio using visual prompts, requiring a deep understanding of spatial and semantic information. However, current models risk overfitting to room environments and lose fine-grained spatial details. In this paper, we propose a new audio-visual binaural generation model incorporating an audio-visual conditional normalisation layer that dynamically aligns the mean and variance of the target difference audio features using visual context, along with a new contrastive learning method to enhance spatial sensitivity by mining negative samples from shuffled visual features. We also introduce a cost-efficient way to utilise test-time augmentation in video data to enhance performance. Our approach achieves state-of-the-art generation accuracy on the FAIR-Play and MUSIC-Stereo benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
