arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.08490cs.CVcs.MMcs.SDeess.AS
收藏

通过拟音类比实现视频到音频的条件生成

Conditional Generation of Audio from Video via Foley Analogies

University of Michigan(密歇根大学);Yale University(耶鲁大学);Adobe Research(奥多比研究院)

Yuexi Du, Ziyang Chen, Justin Salamon, Bryan Russell, Andrew Owens

AI总结:

本文提出条件拟音问题,通过从同一视频中采样条件音视频片段的预训练任务和基于用户示例的条件生成模型,实现根据指定示例为无声视频生成可变音效。

摘要:

设计师为视频添加的音效旨在传达特定的艺术效果,因此可能与场景的真实声音存在很大差异。受为视频创作与其真实声音不同但仍与屏幕上发生的动作相匹配的音轨这一挑战的启发,我们提出了条件拟音(conditional Foley)问题。为解决该问题,我们做出以下贡献。首先,我们提出一个预训练任务,利用从同一源视频中另一时间采样的条件音视频片段,训练模型为输入视频片段预测声音。其次,我们提出一个模型,在给定用户提供的示例来指定视频应该“听起来像什么”的情况下,为无声输入视频生成音轨。我们通过人工研究和自动化评估指标表明,我们的模型能够成功地从视频生成声音,同时根据所提供示例的内容改变其输出。项目网站:https://xypb.github.io/CondFoleyGen/

英文摘要:

The sound effects that designers add to videos are designed to convey a particular artistic effect and, thus, may be quite different from a scene's true sound. Inspired by the challenges of creating a soundtrack for a video that differs from its true sound, but that nonetheless matches the actions occurring on screen, we propose the problem of conditional Foley. We present the following contributions to address this problem. First, we propose a pretext task for training our model to predict sound for an input video clip using a conditional audio-visual clip sampled from another time within the same source video. Second, we propose a model for generating a soundtrack for a silent input video, given a user-supplied example that specifies what the video should "sound like". We show through human studies and automated evaluation metrics that our model successfully generates sound from video, while varying its output according to the content of a supplied example. Project site: https://xypb.github.io/CondFoleyGen/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑