VFX Creator：基于可控 Diffusion Transformer 的动画视觉特效生成
VFX Creator: Animated Visual Effect Generation with Controllable Diffusion Transformer
AI总结：
本文提出VFX Creator框架，通过Video Diffusion Transformer和Open-VFX数据集，利用可控LoRA适配器、掩码控制模块及时间戳实现精准的时空视觉特效生成。
摘要：
打造魔法与幻象是电影制作中最激动人心的方面之一，而视觉特效（VFX）是令人难忘的电影体验背后的强大驱动力。尽管生成式人工智能的最新进展推动了通用图像和视频合成的进步，但可控 VFX 生成领域仍然相对缺乏探索。在这项工作中，我们提出了一种将动画 VFX 生成作为图像动画的新范式，其中动态特效是根据用户友好的文本描述和静态参考图像生成的。我们的工作做出了两项主要贡献：(i) Open-VFX，这是首个涵盖 15 个不同特效类别的高质量 VFX 视频数据集，标注了文本描述、用于空间条件的实例分割掩码，以及用于时间控制的起止时间戳。(ii) VFX Creator，一个基于 Video Diffusion Transformer 的简单而有效的可控 VFX 生成框架。该模型结合了空间和时间可控的 LoRA 适配器，只需极少的训练视频。具体而言，一个即插即用的掩码控制模块实现了实例级的空间操作，而嵌入到扩散过程中并与文本编码器配合的标记化起止运动时间戳，允许对特效的时机和节奏进行精确的时间控制。在 Open-VFX 测试集上的大量实验证明了所提出系统在生成逼真动态特效方面的优越性，在空间和时间可控性方面均实现了最先进的性能和泛化能力。此外，我们还引入了一个专门的指标来评估时间控制的精度。通过将传统 VFX 技术与生成方法相结合，VFX Creator 为高效、高质量的视频特效生成解锁了新的可能性，使高级 VFX 能够为更广泛的受众所使用。
英文摘要：
Crafting magic and illusions is one of the most thrilling aspects of filmmaking, with visual effects (VFX) serving as the powerhouse behind unforgettable cinematic experiences. While recent advances in generative artificial intelligence have driven progress in generic image and video synthesis, the domain of controllable VFX generation remains relatively underexplored. In this work, we propose a novel paradigm for animated VFX generation as image animation, where dynamic effects are generated from user-friendly textual descriptions and static reference images. Our work makes two primary contributions: (i) Open-VFX, the first high-quality VFX video dataset spanning 15 diverse effect categories, annotated with textual descriptions, instance segmentation masks for spatial conditioning, and start-end timestamps for temporal control. (ii) VFX Creator, a simple yet effective controllable VFX generation framework based on a Video Diffusion Transformer. The model incorporates a spatial and temporal controllable LoRA adapter, requiring minimal training videos. Specifically, a plug-and-play mask control module enables instance-level spatial manipulation, while tokenized start-end motion timestamps embedded in the diffusion process, alongside the text encoder, allow precise temporal control over effect timing and pace. Extensive experiments on the Open-VFX test set demonstrate the superiority of the proposed system in generating realistic and dynamic effects, achieving state-of-the-art performance and generalization ability in both spatial and temporal controllability. Furthermore, we introduce a specialized metric to evaluate the precision of temporal control. By bridging traditional VFX techniques with generative approaches, VFX Creator unlocks new possibilities for efficient and high-quality video effect generation, making advanced VFX accessible to a broader audience.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
