VidToMe: 面向零样本视频编辑的视频令牌合并
VidToMe: Video Token Merging for Zero-Shot Video Editing
AI总结：
针对零样本视频编辑中时间一致性难保障、内存效率低的问题，提出VidToMe方法，通过跨帧自注意力令牌合并及分块的局部-全局合并策略，提升时间一致性并降低内存消耗，效果优于现有最优方法。
摘要：
扩散模型在生成高质量图像方面已取得显著进展，但由于时间运动的复杂性，其在视频生成中的应用仍颇具挑战。零样本视频编辑利用预训练图像扩散模型将源视频转换为新视频，为该问题提供了解决方案。然而，现有方法难以维持严格的时间一致性，也无法实现高效的内存消耗。本研究中，我们提出一种新方法，通过合并跨帧的自注意力令牌来增强生成视频的时间一致性。通过对齐并压缩跨帧的时间冗余令牌，我们的方法提升了时间连贯性，同时降低了自注意力计算中的内存消耗。该合并策略根据帧间的时间对应关系匹配并对齐令牌，助力生成的视频帧实现自然的时间一致性。为管控视频处理的复杂度，我们将视频划分为块，并提出块内局部令牌合并与块间全局令牌合并机制，同时保障短期视频连续性与长期内容一致性。我们的视频编辑方法将图像编辑领域的进展无缝拓展至视频编辑，在时间一致性上取得了优于现有最优方法的结果。
英文摘要：
Diffusion models have made significant advances in generating high-quality images, but their application to video generation has remained challenging due to the complexity of temporal motion. Zero-shot video editing offers a solution by utilizing pre-trained image diffusion models to translate source videos into new ones. Nevertheless, existing methods struggle to maintain strict temporal consistency and efficient memory consumption. In this work, we propose a novel approach to enhance temporal consistency in generated videos by merging self-attention tokens across frames. By aligning and compressing temporally redundant tokens across frames, our method improves temporal coherence and reduces memory consumption in self-attention computations. The merging strategy matches and aligns tokens according to the temporal correspondence between frames, facilitating natural temporal consistency in generated video frames. To manage the complexity of video processing, we divide videos into chunks and develop intra-chunk local token merging and inter-chunk global token merging, ensuring both short-term video continuity and long-term content consistency. Our video editing approach seamlessly extends the advancements in image editing to video editing, rendering favorable results in temporal consistency over state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
