零样本视频翻译 via 标签扭曲
Zero-Shot Video Translation via Token Warping
AI总结:
本文提出TokenWarping框架,通过光流扭曲提升视频翻译的时间一致性与质量。
摘要:
随着生成式AI的革命,视频相关任务已被广泛研究。然而,当前最先进的视频模型在视觉质量和用户对生成内容的控制方面仍落后于图像模型。在本文中,我们介绍了TokenWarping,一种用于时间一致视频翻译的新框架。现有的基于扩散的视频编辑方法仅依赖于自注意力中的键值块来确保时间一致性,往往牺牲了局部和结构区域的保持。关键的是,这些方法忽略了查询块在实现准确特征聚合和时间一致性中的重要性。相反,TokenWarping通过在不同帧之间构建时间相关性来利用互补的token先验。我们的方法首先从源视频中提取光流。在扩散模型的去噪过程中,这些光流被用来扭曲前一帧的查询、键和值块,使其与当前帧的块对齐。通过直接扭曲查询块,我们增强了自注意力中的特征聚合,而扭曲键和值块则确保了帧之间的时间一致性。这种token扭曲对自注意力层的输出施加了显式的约束,从而有效确保了时间一致的翻译。我们的框架不需要任何额外的训练或微调,可以无缝集成到现有的文本到图像编辑方法中。我们在各种视频翻译任务上进行了广泛的实验,证明TokenWarping在定性和定量上都优于最先进的方法。视频演示可在我们的项目网页上找到:https://alex-zhu1.github.io/TokenWarping/。代码可在:https://github.com/Alex-Zhu1/TokenWarping。
英文摘要:
With the revolution of generative AI, video-related tasks have been widely studied. However, current state-of-the-art video models still lag behind image models in visual quality and user control over generated content. In this paper, we introduce TokenWarping, a novel framework for temporally coherent video translation. Existing diffusion-based video editing approaches rely solely on key and value patches in self-attention to ensure temporal consistency, often sacrificing the preservation of local and structural regions. Critically, these methods overlook the significance of the query patches in achieving accurate feature aggregation and temporal coherence. In contrast, TokenWarping leverages complementary token priors by constructing temporal correlations across different frames. Our method begins by extracting optical flows from source videos. During the denoising process of the diffusion model, these optical flows are used to warp the previous frame's query, key, and value patches, aligning them with the current frame's patches. By directly warping the query patches, we enhance feature aggregation in self-attention, while warping the key and value patches ensures temporal consistency across frames. This token warping imposes explicit constraints on the self-attention layer outputs, effectively ensuring temporally coherent translation. Our framework does not require any additional training or fine-tuning and can be seamlessly integrated with existing text-to-image editing methods. We conduct extensive experiments on various video translation tasks, demonstrating that TokenWarping surpasses state-of-the-art methods both qualitatively and quantitatively. Video demonstrations can be found on our project webpage: https://alex-zhu1.github.io/TokenWarping/. Code is available at: https://github.com/Alex-Zhu1/TokenWarping.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。