面向高效视频编辑的以对象为中心的扩散方法
Object-Centric Diffusion for Efficient Video Editing
AI总结：
本文提出Object-Centric Diffusion方法，通过以对象为中心的采样和token合并将计算集中于前景区域，无需重新训练即可将视频编辑延迟降低最高10倍并保持质量。
摘要：
基于扩散的视频编辑已经达到了令人印象深刻的质量，并且能够根据文本编辑提示转换给定视频输入的全局风格、局部结构和属性。然而，此类解决方案通常会产生沉重的内存和计算成本，以生成时间上连贯的帧，无论是以扩散反演和/或跨帧注意力的形式。在本文中，我们对这些低效之处进行了分析，并提出简单而有效的修改，在保持质量的同时实现显著加速。此外，我们引入了Object-Centric Diffusion，通过将更多计算分配给前景编辑区域来修复生成伪影并进一步降低延迟，这对感知质量可能更为重要。我们通过两项新颖的提议实现了这一点：i) Object-Centric Sampling，解耦用于显著区域或背景区域的扩散步骤，并将大部分步骤用于前者；ii) Object-Centric Token Merging，通过在不重要的背景区域中融合冗余token来降低跨帧注意力的成本。这两种技术都无需重新训练即可直接应用于给定的视频编辑模型，并能大幅降低其内存和计算成本。我们在基于反演和基于控制信号的编辑流程上评估了我们的提议，结果表明在可比的合成质量下，延迟降低高达10倍。项目页面：qualcomm-ai-research.github.io/object-centric-diffusion。
英文摘要：
Diffusion-based video editing have reached impressive quality and can transform either the global style, local structure, and attributes of given video inputs, following textual edit prompts. However, such solutions typically incur heavy memory and computational costs to generate temporally-coherent frames, either in the form of diffusion inversion and/or cross-frame attention. In this paper, we conduct an analysis of such inefficiencies, and suggest simple yet effective modifications that allow significant speed-ups whilst maintaining quality. Moreover, we introduce Object-Centric Diffusion, to fix generation artifacts and further reduce latency by allocating more computations towards foreground edited regions, arguably more important for perceptual quality. We achieve this by two novel proposals: i) Object-Centric Sampling, decoupling the diffusion steps spent on salient or background regions and spending most on the former, and ii) Object-Centric Token Merging, which reduces cost of cross-frame attention by fusing redundant tokens in unimportant background regions. Both techniques are readily applicable to a given video editing model without retraining, and can drastically reduce its memory and computational cost. We evaluate our proposals on inversion-based and control-signal-based editing pipelines, and show a latency reduction up to 10x for a comparable synthesis quality. Project page: qualcomm-ai-research.github.io/object-centric-diffusion.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
