基于多模态规划与结构化噪声初始化的文本到视频生成免训练引导
Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initialization
AI总结：
本文提出免训练的文本到视频引导方法Video-MSG，通过多模态规划生成Video Sketch时空草稿，并结合结构化噪声初始化引导扩散模型，在无需微调和额外注意力内存的情况下提升多个T2V骨干网络的文本对齐效果。
摘要：
近年来，文本到视频（T2V）扩散模型的进展显著提升了生成视频的视觉质量。然而，即使是最近的T2V模型也难以准确遵循文本描述，尤其是当提示词要求精确控制空间布局或物体轨迹时。最近的一类研究在T2V模型中使用布局引导，但需要在推理阶段进行微调或对注意力图进行迭代操作。这显著增加了内存需求，使得难以采用大型T2V模型作为骨干网络。为了解决这一问题，我们提出了Video-MSG，一种基于多模态规划和结构化噪声初始化的免训练T2V生成引导方法。Video-MSG包含三个步骤，在前两个步骤中，Video-MSG创建Video Sketch，即以草稿视频帧的形式为最终视频生成细粒度的时空规划，明确指定背景、前景和物体轨迹。在最后一步中，Video-MSG通过噪声反演和去噪，利用Video Sketch引导下游T2V扩散模型。值得注意的是，Video-MSG在推理阶段无需微调或额外内存的注意力操作，从而更容易采用大型T2V模型。Video-MSG在流行的T2V生成基准（T2VCompBench和VBench）上，与多个T2V骨干网络（VideoCrafter2和CogVideoX-5B）结合，展示了其在增强文本对齐方面的有效性。我们提供了关于噪声反演比例、不同背景生成器、背景物体检测和前景物体分割的全面消融研究。
英文摘要：
Recent advancements in text-to-video (T2V) diffusion models have significantly enhanced the visual quality of the generated videos. However, even recent T2V models find it challenging to follow text descriptions accurately, especially when the prompt requires accurate control of spatial layouts or object trajectories. A recent line of research uses layout guidance for T2V models that require fine-tuning or iterative manipulation of the attention map during inference time. This significantly increases the memory requirement, making it difficult to adopt a large T2V model as a backbone. To address this, we introduce Video-MSG, a training-free Guidance method for T2V generation based on Multimodal planning and Structured noise initialization. Video-MSG consists of three steps, where in the first two steps, Video-MSG creates Video Sketch, a fine-grained spatio-temporal plan for the final video, specifying background, foreground, and object trajectories, in the form of draft video frames. In the last step, Video-MSG guides a downstream T2V diffusion model with Video Sketch through noise inversion and denoising. Notably, Video-MSG does not need fine-tuning or attention manipulation with additional memory during inference time, making it easier to adopt large T2V models. Video-MSG demonstrates its effectiveness in enhancing text alignment with multiple T2V backbones (VideoCrafter2 and CogVideoX-5B) on popular T2V generation benchmarks (T2VCompBench and VBench). We provide comprehensive ablation studies about noise inversion ratio, different background generators, background object detection, and foreground object segmentation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
