CustomCrafter：保持运动与概念组合能力的定制化视频生成
CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities
AI总结：
本文提出CustomCrafter框架，通过即插即用主体学习模块和动态加权视频采样策略，在无需额外视频和恢复性微调的情况下，同时保持视频扩散模型的运动生成能力和概念组合能力，显著提升定制化视频生成质量。
摘要：
定制化视频生成旨在根据文本提示和主体的参考图像生成高质量视频。然而，由于主体学习仅在静态图像上进行训练，其微调过程会破坏视频扩散模型（VDMs）组合概念和生成运动的能力。为了恢复这些能力，一些方法使用与提示相似的额外视频来微调或引导模型。这要求在生成不同运动时频繁更换引导视频，甚至需要重新微调模型，对用户来说非常不便。在本文中，我们提出了CustomCrafter，这是一个新颖的框架，无需额外视频和恢复性微调即可保持模型的运动生成能力和概念组合能力。为了保持概念组合能力，我们设计了一个即插即用模块来更新VDMs中的少量参数，增强模型捕捉新主体外观细节的能力以及概念组合能力。对于运动生成，我们观察到VDMs在去噪的早期阶段倾向于恢复视频的运动，而在后期阶段则专注于恢复主体细节。因此，我们提出了动态加权视频采样策略。利用我们主体学习模块的可插拔性，我们在去噪早期阶段减少该模块对运动生成的影响，保持VDMs生成运动的能力。在去噪后期阶段，我们恢复该模块以修复指定主体的外观细节，从而确保主体外观的保真度。实验结果表明，与先前方法相比，我们的方法有显著改进。代码可在 https://github.com/WuTao-CS/CustomCrafter 获取。
英文摘要：
Customized video generation aims to generate high-quality videos guided by text prompts and subject's reference images. However, since it is only trained on static images, the fine-tuning process of subject learning disrupts abilities of video diffusion models (VDMs) to combine concepts and generate motions. To restore these abilities, some methods use additional video similar to the prompt to fine-tune or guide the model. This requires frequent changes of guiding videos and even re-tuning of the model when generating different motions, which is very inconvenient for users. In this paper, we propose CustomCrafter, a novel framework that preserves the model's motion generation and conceptual combination abilities without additional video and fine-tuning to recovery. For preserving conceptual combination ability, we design a plug-and-play module to update few parameters in VDMs, enhancing the model's ability to capture the appearance details and the ability of concept combinations for new subjects. For motion generation, we observed that VDMs tend to restore the motion of video in the early stage of denoising, while focusing on the recovery of subject details in the later stage. Therefore, we propose Dynamic Weighted Video Sampling Strategy. Using the pluggability of our subject learning modules, we reduce the impact of this module on motion generation in the early stage of denoising, preserving the ability to generate motion of VDMs. In the later stage of denoising, we restore this module to repair the appearance details of the specified subject, thereby ensuring the fidelity of the subject's appearance. Experimental results show that our method has a significant improvement compared to previous methods. Code is available at https://github.com/WuTao-CS/CustomCrafter
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
