VideoMaker：利用视频扩散模型固有力量的零样本定制视频生成
VideoMaker: Zero-shot Customized Video Generation with the Inherent Force of Video Diffusion Models
AI总结：
本文提出VideoMaker框架，揭示视频扩散模型本身具备主体特征提取与注入能力，通过内在特征提取和空间自注意力双向交互实现高质量零样本定制视频生成。
摘要：
零样本定制视频生成因其巨大的应用潜力而受到广泛关注。现有方法依赖额外模型来提取和注入参考主体特征，其假设是仅靠视频扩散模型（VDM）不足以完成零样本定制视频生成。然而，由于特征提取和注入技术不够理想，这些方法往往难以保持主体外观的一致性。在本文中，我们揭示了VDM本身固有地具备提取和注入主体特征的力量。不同于以往的启发式方法，我们提出了一种新颖的框架，利用VDM的固有力量来实现高质量的零样本定制视频生成。具体而言，在特征提取方面，我们直接将参考图像输入VDM，并利用其内在的特征提取过程，这不仅提供了细粒度特征，而且与VDM的预训练知识高度对齐。在特征注入方面，我们设计了一种创新的双向交互机制，通过VDM内部的空间自注意力在主体特征与生成内容之间进行交互，确保VDM在保持生成视频多样性的同时具有更好的主体保真度。在定制人物和物体视频生成上的实验验证了我们框架的有效性。
英文摘要：
Zero-shot customized video generation has gained significant attention due to its substantial application potential. Existing methods rely on additional models to extract and inject reference subject features, assuming that the Video Diffusion Model (VDM) alone is insufficient for zero-shot customized video generation. However, these methods often struggle to maintain consistent subject appearance due to suboptimal feature extraction and injection techniques. In this paper, we reveal that VDM inherently possesses the force to extract and inject subject features. Departing from previous heuristic approaches, we introduce a novel framework that leverages VDM's inherent force to enable high-quality zero-shot customized video generation. Specifically, for feature extraction, we directly input reference images into VDM and use its intrinsic feature extraction process, which not only provides fine-grained features but also significantly aligns with VDM's pre-trained knowledge. For feature injection, we devise an innovative bidirectional interaction between subject features and generated content through spatial self-attention within VDM, ensuring that VDM has better subject fidelity while maintaining the diversity of the generated video. Experiments on both customized human and object video generation validate the effectiveness of our framework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
