FluencyVE：将时间感知Mamba与旁路注意力相结合用于视频编辑
FluencyVE: Marrying Temporal-Aware Mamba with Bypass Attention for Video Editing
AI总结：
FluencyVE通过结合时间感知Mamba与旁路注意力，实现高效的视频编辑，减少计算成本并提升生成能力。
摘要：
大规模的文本到图像扩散模型在图像生成和编辑中取得了前所未有的成功。然而，将这种成功扩展到视频编辑仍具挑战性。最近的视频编辑努力已通过添加时间注意力机制来适应预训练的文本到图像模型以处理视频任务。不幸的是，这些方法仍然面临时间不一致的问题和高计算开销。在本研究中，我们提出了FluencyVE，这是一种简单而有效的单次视频编辑方法。FluencyVE将线性时间序列模块Mamba整合到基于预训练Stable Diffusion模型的视频编辑模型中，用以替代时间注意力层。这使模型能够实现全局帧级注意力的同时减少计算成本。此外，我们采用低秩近似矩阵来替代因果注意力中的查询和键权重矩阵，并在训练过程中使用加权平均技术来更新注意力分数。这种方法在显著保留文本到图像模型的生成能力的同时，有效减少了计算负担。实验和分析表明，在现实世界的视频中编辑各种属性、主体和位置方面取得了有希望的结果。
英文摘要：
Large-scale text-to-image diffusion models have achieved unprecedented success in image generation and editing. However, extending this success to video editing remains challenging. Recent video editing efforts have adapted pretrained text-to-image models by adding temporal attention mechanisms to handle video tasks. Unfortunately, these methods continue to suffer from temporal inconsistency issues and high computational overheads. In this study, we propose FluencyVE, which is a simple yet effective one-shot video editing approach. FluencyVE integrates the linear time-series module, Mamba, into a video editing model based on pretrained Stable Diffusion models, replacing the temporal attention layer. This enables global frame-level attention while reducing the computational costs. In addition, we employ low-rank approximation matrices to replace the query and key weight matrices in the causal attention, and use a weighted averaging technique during training to update the attention scores. This approach significantly preserves the generative power of the text-to-image model while effectively reducing the computational burden. Experiments and analyses demonstrate promising results in editing various attributes, subjects, and locations in real-world videos.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
