Still-Moving：无需定制视频数据的定制化视频生成
Still-Moving: Customized Video Generation without Customized Video Data
AI总结：
针对定制化视频生成缺乏对应视频数据的问题，本研究提出Still-Moving框架，借助空间与运动适配器，仅用定制T2I模型的静态图像数据即可实现T2V模型的定制化，在多类生成任务中实现空间与运动先验的无缝融合。
摘要：
文本到图像（T2I）模型的定制化近来已取得巨大进展，尤其在个性化、风格化和条件生成等领域。然而，将这一进展拓展至视频生成仍处于起步阶段，主要原因在于缺乏定制化视频数据。在本研究中，我们提出了Still-Moving，这是一种用于定制文本到视频（T2V）模型的新型通用框架，无需任何定制化视频数据。该框架适用于主流的T2V设计——即视频模型构建于文本到图像（T2I）模型之上（例如通过膨胀方式）。我们假设可以获取仅在静态图像数据上训练的定制化T2I模型（例如使用DreamBooth或StyleDrop训练的模型）。直接将定制化T2I模型的权重接入T2V模型通常会导致严重的伪影，或是对定制数据的贴合度不足。为解决这一问题，我们训练了轻量级的空间适配器（Spatial Adapters），用于调整注入的T2I层生成的特征。关键在于，我们的适配器在“冻结视频”（即重复图像）上训练，而这些视频由定制化T2I模型生成的图像样本构建而成。这一训练过程借助一种新型的运动适配器（Motion Adapter）模块实现，该模块让我们能够在这类静态视频上训练，同时保留视频模型的运动先验。在测试阶段，我们移除运动适配器模块，仅保留训练好的空间适配器。这一操作既恢复了T2V模型的运动先验，又贴合了定制化T2I模型的空间先验。我们在个性化生成、风格化生成、条件生成等多种任务上验证了该方法的有效性。在所有评估场景中，我们的方法都能将定制化T2I模型的空间先验与T2V模型提供的运动先验无缝融合。
英文摘要：
Customizing text-to-image (T2I) models has seen tremendous progress recently, particularly in areas such as personalization, stylization, and conditional generation. However, expanding this progress to video generation is still in its infancy, primarily due to the lack of customized video data. In this work, we introduce Still-Moving, a novel generic framework for customizing a text-to-video (T2V) model, without requiring any customized video data. The framework applies to the prominent T2V design where the video model is built over a text-to-image (T2I) model (e.g., via inflation). We assume access to a customized version of the T2I model, trained only on still image data (e.g., using DreamBooth or StyleDrop). Naively plugging in the weights of the customized T2I model into the T2V model often leads to significant artifacts or insufficient adherence to the customization data. To overcome this issue, we train lightweight $\textit{Spatial Adapters}$ that adjust the features produced by the injected T2I layers. Importantly, our adapters are trained on $\textit{"frozen videos"}$ (i.e., repeated images), constructed from image samples generated by the customized T2I model. This training is facilitated by a novel $\textit{Motion Adapter}$ module, which allows us to train on such static videos while preserving the motion prior of the video model. At test time, we remove the Motion Adapter modules and leave in only the trained Spatial Adapters. This restores the motion prior of the T2V model while adhering to the spatial prior of the customized T2I model. We demonstrate the effectiveness of our approach on diverse tasks including personalized, stylized, and conditional generation. In all evaluated scenarios, our method seamlessly integrates the spatial prior of the customized T2I model with a motion prior supplied by the T2V model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
