arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.09195cs.CV
收藏

基于预训练扩散模型的免训练语义视频合成

Training-Free Semantic Video Composition via Pre-trained Diffusion Model

University of Electronic Science and Technology of China (UESTC)(电子科技大学)

Jiaqi Guo, Sitong Su, Junchen Zhu, Lianli Gao, Jingkuan Song

AI总结:

本文提出一种免训练的语义视频合成流程,利用预训练扩散模型,通过平衡部分反演和帧间增强注意力机制,有效处理前景与背景之间的深层语义差异,并保证输出视频的视觉和谐与帧间连贯性。

摘要:

视频合成任务旨在将来自不同视频的指定前景和背景整合为一个和谐的整体。当前方法主要基于对前景颜色和光照进行调整后的视频进行训练,难以解决超出表面调整之外的深层语义差异,例如域差距。因此,我们提出了一种免训练流程,采用具有语义先验知识的预训练扩散模型,能够处理具有更广泛语义差异的合成视频。具体而言,我们以级联方式处理视频帧,并利用扩散模型在两个过程中处理每一帧。在反演过程中,我们提出平衡部分反演(Balanced Partial Inversion),以获得平衡可逆性与可修改性的生成初始点。然后,在生成过程中,我们进一步提出帧间增强注意力(Inter-Frame Augmented attention),以增强前景在帧间的连续性。实验结果表明,我们的流程成功确保了输出的视觉和谐性与帧间连贯性,证明了其在处理更广泛语义差异方面的有效性。

英文摘要:

The video composition task aims to integrate specified foregrounds and backgrounds from different videos into a harmonious composite. Current approaches, predominantly trained on videos with adjusted foreground color and lighting, struggle to address deep semantic disparities beyond superficial adjustments, such as domain gaps. Therefore, we propose a training-free pipeline employing a pre-trained diffusion model imbued with semantic prior knowledge, which can process composite videos with broader semantic disparities. Specifically, we process the video frames in a cascading manner and handle each frame in two processes with the diffusion model. In the inversion process, we propose Balanced Partial Inversion to obtain generation initial points that balance reversibility and modifiability. Then, in the generation process, we further propose Inter-Frame Augmented attention to augment foreground continuity across frames. Experimental results reveal that our pipeline successfully ensures the visual harmony and inter-frame coherence of the outputs, demonstrating efficacy in managing broader semantic disparities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑