ShotAdapter：基于扩散模型的文本到多镜头视频生成
ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models
AI总结：
本文提出 ShotAdapter 框架，通过转场 token、局部注意力掩码和多镜头数据流水线扩展视频扩散模型，实现角色与背景一致且可控的文本到多镜头视频生成。
摘要：
当前基于扩散模型的文本到视频方法仅限于生成单镜头短视频片段，缺乏生成具有离散转场的多镜头视频的能力；在这类视频中，同一角色会在相同或不同背景下执行不同活动。为解决这一局限，我们提出一个框架，其中包含数据集收集流水线以及对视频扩散模型的架构扩展，以实现文本到多镜头视频生成。我们的方法能够将多镜头视频生成为单个视频，并在所有镜头的所有帧之间使用完整注意力，从而确保角色和背景的一致性，同时允许用户通过镜头特定条件控制镜头的数量、持续时间和内容。这是通过在文本到视频模型中引入转场 token 来控制新镜头从哪些帧开始，并采用局部注意力掩码策略来控制转场 token 的作用范围以及支持镜头特定提示实现的。为获取训练数据，我们提出一种新颖的数据收集流水线，可从现有单镜头视频数据集构建多镜头视频数据集。大量实验表明，将预训练文本到视频模型微调数千次迭代，就足以使模型随后生成具有镜头特定控制的多镜头视频，并且性能优于基线方法。更多细节可参见 https://shotadapter.github.io/。
英文摘要：
Current diffusion-based text-to-video methods are limited to producing short video clips of a single shot and lack the capability to generate multi-shot videos with discrete transitions where the same character performs distinct activities across the same or different backgrounds. To address this limitation we propose a framework that includes a dataset collection pipeline and architectural extensions to video diffusion models to enable text-to-multi-shot video generation. Our approach enables generation of multi-shot videos as a single video with full attention across all frames of all shots, ensuring character and background consistency, and allows users to control the number, duration, and content of shots through shot-specific conditioning. This is achieved by incorporating a transition token into the text-to-video model to control at which frames a new shot begins and a local attention masking strategy which controls the transition token's effect and allows shot-specific prompting. To obtain training data we propose a novel data collection pipeline to construct a multi-shot video dataset from existing single-shot video datasets. Extensive experiments demonstrate that fine-tuning a pre-trained text-to-video model for a few thousand iterations is enough for the model to subsequently be able to generate multi-shot videos with shot-specific control, outperforming the baselines. You can find more details in https://shotadapter.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
