Magic-Me：身份特定的视频定制扩散
Magic-Me: Identity-Specific Video Customized Diffusion
AI总结：
提出Video Custom Diffusion框架，通过3D高斯噪声初始化、ID模块及Face/Tiled VCD模块，实现身份可控的稳定视频生成，并支持与公开T2I模型兼容。
摘要：
在生成模型领域，创建具有指定身份（ID）的内容引起了广泛关注。在文本到图像生成（T2I）领域，通过参考图像控制身份的主体驱动创作取得了巨大进展。然而，其向视频生成的扩展尚未得到充分探索。在这项工作中，我们提出了一个简单而有效的主体身份可控视频生成框架，称为Video Custom Diffusion（VCD）。通过由少量图像定义的指定身份，VCD在初始化阶段强化身份特征并注入逐帧相关性，以实现稳定的视频输出。为此，我们提出了三个对高质量身份保持和稳定视频生成至关重要的新颖组件：1）一种具有3D高斯噪声先验的噪声初始化方法，以实现更好的帧间稳定性；2）基于扩展Textual Inversion的ID模块，使用裁剪的身份进行训练，将ID信息与背景解耦；3）Face VCD和Tiled VCD模块，用于强化人脸并将视频放大到更高分辨率，同时保持身份特征。我们进行了大量实验，验证了VCD能够生成比基线具有更好ID的稳定视频。此外，凭借ID模块中编码身份的可迁移性，VCD也能与公开可用的个性化文本到图像模型良好配合。代码可在https://github.com/Zhen-Dong/Magic-Me获取。
英文摘要：
Creating content with specified identities (ID) has attracted significant interest in the field of generative models. In the field of text-to-image generation (T2I), subject-driven creation has achieved great progress with the identity controlled via reference images. However, its extension to video generation is not well explored. In this work, we propose a simple yet effective subject identity controllable video generation framework, termed Video Custom Diffusion (VCD). With a specified identity defined by a few images, VCD reinforces the identity characteristics and injects frame-wise correlation at the initialization stage for stable video outputs. To achieve this, we propose three novel components that are essential for high-quality identity preservation and stable video generation: 1) a noise initialization method with 3D Gaussian Noise Prior for better inter-frame stability; 2) an ID module based on extended Textual Inversion trained with the cropped identity to disentangle the ID information from the background 3) Face VCD and Tiled VCD modules to reinforce faces and upscale the video to higher resolution while preserving the identity's features. We conducted extensive experiments to verify that VCD is able to generate stable videos with better ID over the baselines. Besides, with the transferability of the encoded identity in the ID module, VCD is also working well with personalized text-to-image models available publicly. The codes are available at https://github.com/Zhen-Dong/Magic-Me.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
