LayerFlow：一个面向分层感知视频生成的统一模型
LayerFlow: A Unified Model for Layer-aware Video Generation
AI总结：
LayerFlow提出统一框架，利用层嵌入和子片段组织方式，结合多阶段训练策略与LoRA微调，实现分层感知视频生成及多种变体任务。
摘要：
我们提出LayerFlow，一个用于分层感知视频生成的统一解决方案。给定逐层提示，LayerFlow为透明前景、干净背景和混合场景生成视频。它还支持多种变体，例如分解混合视频，或为给定前景生成背景，反之亦然。从一个文本到视频的扩散Transformer出发，我们将不同层的视频组织为子片段，并利用层嵌入来区分每个片段及其对应的逐层提示。通过这种方式，我们在一个统一框架中无缝支持上述变体。由于缺乏高质量的逐层训练视频，我们设计了一种多阶段训练策略，以适应具有高质量分层标注的静态图像。具体而言，我们首先使用低质量视频数据训练模型。然后，我们微调一个运动LoRA，使模型兼容静态帧。之后，我们在混合图像数据上训练内容LoRA，这些数据包括高质量分层图像以及复制粘贴的视频数据。在推理过程中，我们移除运动LoRA，从而生成具有所需层次的平滑视频。
英文摘要：
We present LayerFlow, a unified solution for layer-aware video generation. Given per-layer prompts, LayerFlow generates videos for the transparent foreground, clean background, and blended scene. It also supports versatile variants like decomposing a blended video or generating the background for the given foreground and vice versa. Starting from a text-to-video diffusion transformer, we organize the videos for different layers as sub-clips, and leverage layer embeddings to distinguish each clip and the corresponding layer-wise prompts. In this way, we seamlessly support the aforementioned variants in one unified framework. For the lack of high-quality layer-wise training videos, we design a multi-stage training strategy to accommodate static images with high-quality layer annotations. Specifically, we first train the model with low-quality video data. Then, we tune a motion LoRA to make the model compatible with static frames. Afterward, we train the content LoRA on the mixture of image data with high-quality layered images along with copy-pasted video data. During inference, we remove the motion LoRA thus generating smooth videos with desired layers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
