AudioScenic：音频驱动的视频场景编辑
AudioScenic: Audio-Driven Video Scene Editing
AI总结：
本文提出AudioScenic框架，通过时间感知音频语义注入、SceneMasker模块及音频幅度与频率引导，实现音频驱动的视频背景编辑，在保持前景完整的同时增强视觉动态与时间一致性。
摘要：
音频驱动的视觉场景编辑致力于根据给定的音频信号，在保持前景内容不变的情况下操纵视觉背景。与当前主要集中于图像编辑的研究不同，音频驱动的视频场景编辑尚未得到广泛探讨。在本文中，我们提出了AudioScenic，一个专为视频场景编辑设计的音频驱动框架。AudioScenic通过时间感知的音频语义注入过程将音频语义整合到视觉场景中。由于我们的重点是背景编辑，我们进一步引入了SceneMasker模块，以在编辑过程中保持前景内容的完整性。AudioScenic利用音频的固有属性，即音频幅度和频率，来指导编辑过程，旨在控制时间动态并增强时间一致性。首先，我们提出了音频幅度调制器模块，该模块根据音频幅度的变化调整场景的时间动态，从而增强视觉动态。其次，音频频率融合器模块旨在通过将音频的频率与视频场景的动态对齐来确保时间一致性，从而提高编辑视频的整体时间连贯性。这些集成特性使AudioScenic不仅能够增强视觉多样性，还能在整个视频中保持时间一致性。我们提出了一种名为时间分数的新指标，以更全面地验证时间一致性。我们在DAVIS和Audioset数据集上展示了AudioScenic相较于竞争方法的实质性进步。
英文摘要：
Audio-driven visual scene editing endeavors to manipulate the visual background while leaving the foreground content unchanged, according to the given audio signals. Unlike current efforts focusing primarily on image editing, audio-driven video scene editing has not been extensively addressed. In this paper, we introduce AudioScenic, an audio-driven framework designed for video scene editing. AudioScenic integrates audio semantics into the visual scene through a temporal-aware audio semantic injection process. As our focus is on background editing, we further introduce a SceneMasker module, which maintains the integrity of the foreground content during the editing process. AudioScenic exploits the inherent properties of audio, namely, audio magnitude and frequency, to guide the editing process, aiming to control the temporal dynamics and enhance the temporal consistency. First, we present an audio Magnitude Modulator module that adjusts the temporal dynamics of the scene in response to changes in audio magnitude, enhancing the visual dynamics. Second, the audio Frequency Fuser module is designed to ensure temporal consistency by aligning the frequency of the audio with the dynamics of the video scenes, thus improving the overall temporal coherence of the edited videos. These integrated features enable AudioScenic to not only enhance visual diversity but also maintain temporal consistency throughout the video. We present a new metric named temporal score for more comprehensive validation of temporal consistency. We demonstrate substantial advancements of AudioScenic over competing methods on DAVIS and Audioset datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
