通过解耦身份和运动实现主体驱动的视频生成
Learning Zero-Shot Subject-Driven Video Generation Using 1% Compute
AI总结：
本文提出一种通过解耦身份和运动实现主体驱动视频生成的方法，在零样本条件下优于现有模型。
摘要：
我们提出通过在零样本情况下解耦主体特定学习与时间动态来训练一个主体驱动的定制视频生成模型。传统视频定制方法通常依赖于大规模标注视频数据集，这些数据集计算成本高且需要大量标注。与之前的方法不同，我们直接在训练视频定制模型时使用图像定制数据集，将视频定制分解为两个部分：（1）通过图像定制数据集进行身份注入，（2）通过图像到视频训练方法保留时间建模。此外，我们在图像到视频微调过程中采用随机图像标记丢弃和随机图像初始化，以缓解复制粘贴问题。为了进一步增强学习，我们引入了在主体特定和时间特征联合优化过程中进行随机切换，以缓解灾难性遗忘。我们的方法在主体一致性和可扩展性方面表现出色，在零样本设置中优于现有的视频定制模型，证明了我们框架的有效性。
英文摘要：
Subject-driven video generation (SDV-Gen) aims to produce videos of a specific subject by adapting a pretrained video model, enabling personalized and application-driven content creation. To achieve this goal, per-subject tuning methods require approximately 200 A100 GPU hours to generate a customized video, whereas zero-shot methods avoid per-subject tuning but typically rely on millions of subject-video pairs for the supervision, incurring massive network fine-tuning costs (10K-200K A100 GPU hours). We propose a data- and compute-efficient zero-shot SDV-Gen framework that avoids test-time per-subject tuning and the use of large-scale subject-video pairs. Our key idea decomposes SDV-Gen into (i) identity injection learned from subject-image pairs and (ii) motion-awareness preservation maintained by a small set of arbitrary videos. We optimize the two tasks with stochastic switching, using random reference-frame sampling and image-token dropout to prevent trivial first-frame copying. Our gradient analysis shows that the two objectives rapidly evolve toward nearly orthogonal update subspaces, explaining the stable optimization. Using CogVideoX-5B, we adapt a single model with 200K subject-image pairs and 4,000 arbitrary videos in 288 A100 GPU hours. This yields about 1% of compute compared to prior zero-shot baselines (i.e., 0.4% of VACE and 2.8% of Phantom) while using no subject-video pairs, yet remaining competitive in subject fidelity and motion quality. We show that the same recipe transfers to Wan 2.1-1.3B and Wan 2.2-5B.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
