Free-Bloom：基于 LLM 导演与 LDM 动画师的零样本文本到视频生成器
Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator
AI总结：
本文提出零样本 Free-Bloom 文本到视频流程，用 LLM 规划连贯提示、LDM 生成高保真帧，并通过联合噪声采样等机制保证时序与身份一致。
摘要：
文本到视频是一个快速发展的研究领域，旨在生成语义一致、身份相同且时间连贯的帧序列，并使其与输入文本提示准确对齐。本研究聚焦于零样本文本到视频生成，以兼顾数据与成本效率。为了生成语义连贯的视频，丰富呈现诸如花朵盛开全过程这类时间语义，而非一组“运动图像”，我们提出了一种新颖的 Free-Bloom 流程：利用大语言模型（LLM）担任导演，生成语义连贯的提示序列；同时让预训练潜在扩散模型（LDM）担任动画师，生成高保真帧。此外，为在保持语义连贯的同时确保时间与身份一致性，我们提出了一系列注释性修改，以在反向过程中适配 LDM，包括联合噪声采样、步骤感知注意力偏移和双路径插值。无需任何视频数据或训练要求，Free-Bloom 即可生成生动且高质量的视频，在生成具有语义意义帧序列的复杂场景方面表现令人惊叹。此外，Free-Bloom 天然兼容基于 LDM 的扩展。
英文摘要：
Text-to-video is a rapidly growing research area that aims to generate a semantic, identical, and temporal coherence sequence of frames that accurately align with the input text prompt. This study focuses on zero-shot text-to-video generation considering the data- and cost-efficient. To generate a semantic-coherent video, exhibiting a rich portrayal of temporal semantics such as the whole process of flower blooming rather than a set of "moving images", we propose a novel Free-Bloom pipeline that harnesses large language models (LLMs) as the director to generate a semantic-coherence prompt sequence, while pre-trained latent diffusion models (LDMs) as the animator to generate the high fidelity frames. Furthermore, to ensure temporal and identical coherence while maintaining semantic coherence, we propose a series of annotative modifications to adapting LDMs in the reverse process, including joint noise sampling, step-aware attention shift, and dual-path interpolation. Without any video data and training requirements, Free-Bloom generates vivid and high-quality videos, awe-inspiring in generating complex scenes with semantic meaningful frame sequences. In addition, Free-Bloom is naturally compatible with LDMs-based extensions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
