arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2008.00820cs.CVcs.SDeess.AS
收藏

从视频中生成视觉对齐的声音

Generating Visually Aligned Sound from Videos

South China University of Technology(华南理工大学);Pazhou Laboratory(琶洲实验室);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室)

Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, Chuang Gan

AI总结:

针对自然视频生成声音易受无关声音干扰、对齐度低的问题,提出REGNET框架,通过音频前向正则化器增强训练监督,显著提升声音的时间与内容对齐效果,生成声音骗过人类的成功率达68.12%。

摘要:

我们聚焦于从自然视频生成声音的任务,生成的声音需在时间和内容上均与视觉信号对齐。该任务极具挑战性,因为部分在摄像头外产生的声音无法从视频内容中推断出来,模型可能被迫学习视觉内容与这些无关声音之间的错误映射关系。为解决这一挑战,我们提出了名为REGNET的框架。在该框架中,我们首先从视频帧中提取外观与运动特征,以更好地从复杂背景信息中区分出发声物体。随后我们引入了一种创新的音频前向正则化器,它直接将真实声音作为输入并输出瓶颈声音特征。训练过程中同时使用视觉特征与瓶颈声音特征进行声音预测,可为声音预测提供更强的监督信号。该音频前向正则化器能够控制无关声音成分,从而防止模型学习到视频帧与屏幕外物体发出的声音之间的错误映射。测试阶段会移除音频前向正则化器,以确保REGNET仅依靠视觉特征就能生成完全对齐的声音。基于Amazon Mechanical Turk的大量评估表明,我们的方法在时间对齐和内容对齐两方面均有显著提升。值得注意的是,我们生成的声音能以68.12%的成功率骗过人类。代码与预训练模型已公开发布于https://github.com/PeihaoChen/regnet。

英文摘要:

We focus on the task of generating sound from natural videos, and the sound should be both temporally and content-wise aligned with visual signals. This task is extremely challenging because some sounds generated \emph{outside} a camera can not be inferred from video content. The model may be forced to learn an incorrect mapping between visual content and these irrelevant sounds. To address this challenge, we propose a framework named REGNET. In this framework, we first extract appearance and motion features from video frames to better distinguish the object that emits sound from complex background information. We then introduce an innovative audio forwarding regularizer that directly considers the real sound as input and outputs bottlenecked sound features. Using both visual and bottlenecked sound features for sound prediction during training provides stronger supervision for the sound prediction. The audio forwarding regularizer can control the irrelevant sound component and thus prevent the model from learning an incorrect mapping between video frames and sound emitted by the object that is out of the screen. During testing, the audio forwarding regularizer is removed to ensure that REGNET can produce purely aligned sound only from visual features. Extensive evaluations based on Amazon Mechanical Turk demonstrate that our method significantly improves both temporal and content-wise alignment. Remarkably, our generated sound can fool the human with a 68.12% success rate. Code and pre-trained models are publicly available at https://github.com/PeihaoChen/regnet

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑