流式视频编辑的简易适配

中文摘要 AI 辅助

在本文中，我们提出了SVEET框架，该框架仅需在预训练的双向视频扩散模型上进行训练，即可支持自回归方式的高质量流式视频编辑。为解决这一问题，我们首先系统地回顾了现有的视频到视频扩散方法，并确定了此类流式适配的两个关键原则：骨干特征解耦和条件帧独立性。基于这些见解，我们开发了一种用于可控视频生成的新范式。其核心是一个辅助模型分支，通过时间独立的自我注意力编码源视频输入，并将中间特征注入相应的骨干块以实现流式兼容的控制。此外，为弥合双向模型与流式模型特征空间之间的差异，我们提出了一种解耦训练方案，明确强制视频可控性与模型因果性优化方向之间的正交性。这种解耦确保了推理时两个目标之间的兼容性，并促进了跨异构骨干架构的平滑零样本知识迁移。大量实验表明，SVEET在保持实时性能的同时实现了卓越的编辑质量，在单个H100 GPU上达到15 FPS，无需任何辅助加速技术。代码可在该https URL获取。

英文摘要

In this paper, we propose SVEET, a framework that requires merely training on a pretrained bidirectional video diffusion model but supports high-quality streaming video editing in an auto-regressive fashion. To tackle this problem, we first systematically revisit existing video-to-video diffusion approaches and identify two key principles for such streaming adaptation: backbone feature disentanglement and conditional frame independence. Building on these insights, we develop a novel paradigm for controllable video generation. At its core, an auxiliary model branch encodes source video inputs with temporally independent self-attention, and the intermediate features are injected into the corresponding backbone blocks for streaming-compatible control. Moreover, to bridge the discrepancy between the feature spaces of bidirectional and streaming models, we propose a decoupled training scheme that explicitly enforces the orthogonality between the optimization directions of video controllability and model causality. Such disentanglement ensures compatibility between the two objectives at inference and facilitates smooth zero-shot knowledge transfer across heterogeneous backbone architectures. Extensive experiments demonstrate that SVEET achieves superior editing quality while maintaining real-time performance, attaining 15 FPS on a single H100 GPU 17 without any auxiliary acceleration techniques. Codes are available at https://github.com/YujiaHu1109/SVEET.

发表机构

• Shanghai Jiao Tong University（上海交通大学）

机构由 AI 辅助整理，请以论文原文为准。