Omni-Effects: 统一且空间可控的视觉效果生成
Omni-Effects: Unified and Spatially-Controllable Visual Effects Generation
AI总结：
Omni-Effects通过统一框架实现空间可控的多效果生成，结合LoRA-MoE和SAP技术，提升视觉效果生成的精度和多样性。
摘要：
视觉效果（VFX）是现代电影制作中至关重要的视觉增强技术。尽管视频生成模型为VFX制作提供了成本效益的解决方案，但当前方法受限于每个效果的LoRA训练，限制了生成单一效果。这种根本限制阻碍了需要空间可控复合效果的应用，即在指定位置同时生成多个效果。然而，将多样化效果整合到统一框架中面临重大挑战：效果变化的干扰和多VFX联合训练中的空间不可控性。为解决这些挑战，我们提出了Omni-Effects，首个能够生成提示引导效果和空间可控复合效果的统一框架。该框架的核心包含两个关键创新：（1）基于LoRA的专家混合（LoRA-MoE），利用一组专家LoRAs，在统一模型中整合多样化效果，同时有效缓解跨任务干扰。（2）空间感知提示（SAP）将空间掩码信息融入文本令牌，实现精确的空间控制。此外，我们引入了一个独立信息流（IIF）模块，整合在SAP中，隔离与单个效果对应的控制信号，以防止任何不期望的混合。为了促进这项研究，我们通过结合图像编辑和首尾帧到视频（FLF2V）合成的新型数据收集管道，构建了全面的VFX数据集Omni-VFX，并引入了专用的VFX评估框架以验证模型性能。大量实验表明，Omni-Effects实现了精确的空间控制和多样化效果生成，使用户能够指定所需效果的类别和位置。
英文摘要：
Visual effects (VFX) are essential visual enhancements fundamental to modern cinematic production. Although video generation models offer cost-efficient solutions for VFX production, current methods are constrained by per-effect LoRA training, which limits generation to single effects. This fundamental limitation impedes applications that require spatially controllable composite effects, i.e., the concurrent generation of multiple effects at designated locations. However, integrating diverse effects into a unified framework faces major challenges: interference from effect variations and spatial uncontrollability during multi-VFX joint training. To tackle these challenges, we propose Omni-Effects, a first unified framework capable of generating prompt-guided effects and spatially controllable composite effects. The core of our framework comprises two key innovations: (1) LoRA-based Mixture of Experts (LoRA-MoE), which employs a group of expert LoRAs, integrating diverse effects within a unified model while effectively mitigating cross-task interference. (2) Spatial-Aware Prompt (SAP) incorporates spatial mask information into the text token, enabling precise spatial control. Furthermore, we introduce an Independent-Information Flow (IIF) module integrated within the SAP, isolating the control signals corresponding to individual effects to prevent any unwanted blending. To facilitate this research, we construct a comprehensive VFX dataset Omni-VFX via a novel data collection pipeline combining image editing and First-Last Frame-to-Video (FLF2V) synthesis, and introduce a dedicated VFX evaluation framework for validating model performance. Extensive experiments demonstrate that Omni-Effects achieves precise spatial control and diverse effect generation, enabling users to specify both the category and location of desired effects.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
