Action2Sound：基于第一人称视频的动作声音环境感知生成
Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos
AI总结：
针对视频与音频弱对应导致生成不受控环境音或幻觉的问题，提出环境感知音频生成模型AV-LDM，通过解耦前景动作音与环境音并利用检索增强生成，在Ego4D等数据集上实现了与视觉内容语义和时间匹配的可控音频生成。
摘要：
为人类动作生成逼真音频对许多应用很重要，如为电影或虚拟现实游戏制作音效。现有方法在训练时隐式假设视频与音频完全对应，但许多声音发生在画面外，与视觉内容仅有微弱甚至无对应关系——导致测试时出现不受控的环境声音或幻觉。我们提出了一种新颖的环境感知音频生成模型AV-LDM。我们设计了一种新颖的音频条件机制，以在野外训练视频中学习将前景动作声音与环境背景声音解耦。给定一个新颖的无声视频，我们的模型使用检索增强生成来创建在语义和时间上均与视觉内容匹配的音频。我们在两个野外第一人称视频数据集Ego4D和EPIC-KITCHENS上训练和评估我们的模型，并引入了Ego4D-Sounds——包含120万个具有动作-音频对应关系的精选片段。我们的模型优于一系列现有方法，允许对环境声音进行可控生成，甚至显示出对计算机图形游戏片段泛化的前景。总体而言，我们的方法是首个尽管使用带有自然背景声音的非精选片段进行训练，仍能将视频到音频生成忠实地聚焦于观察到的视觉内容的方法。
英文摘要：
Generating realistic audio for human actions is important for many applications, such as creating sound effects for films or virtual reality games. Existing approaches implicitly assume total correspondence between the video and audio during training, yet many sounds happen off-screen and have weak to no correspondence with the visuals -- resulting in uncontrolled ambient sounds or hallucinations at test time. We propose a novel ambient-aware audio generation model, AV-LDM. We devise a novel audio-conditioning mechanism to learn to disentangle foreground action sounds from the ambient background sounds in in-the-wild training videos. Given a novel silent video, our model uses retrieval-augmented generation to create audio that matches the visual content both semantically and temporally. We train and evaluate our model on two in-the-wild egocentric video datasets, Ego4D and EPIC-KITCHENS, and we introduce Ego4D-Sounds -- 1.2M curated clips with action-audio correspondence. Our model outperforms an array of existing methods, allows controllable generation of the ambient sound, and even shows promise for generalizing to computer graphics game clips. Overall, our approach is the first to focus video-to-audio generation faithfully on the observed visual content despite training from uncurated clips with natural background sounds.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
