VACE：一体化视频创作与编辑
VACE: All-in-One Video Creation and Editing
AI总结：
VACE提出一体化视频创作与编辑框架，通过视频条件单元和上下文适配器统一参考生成、视频编辑与掩码编辑等任务，性能与专用模型相当并支持灵活任务组合。
摘要：
扩散Transformer在生成高质量图像和视频方面展现出强大的能力与可扩展性。进一步追求生成与编辑任务的统一，已在图像内容创作领域取得显著进展。然而，由于视频合成对时间与空间动态一致性的内在要求，实现统一的视频合成方法仍然具有挑战性。我们提出VACE，使用户能够在一个一体化框架内完成视频创作与编辑任务。这些任务包括参考到视频生成、视频到视频编辑以及掩码视频到视频编辑。具体而言，我们通过将视频任务输入（如编辑、参考和掩码）组织到一个称为视频条件单元（VCU）的统一接口中，有效整合了不同任务的需求。此外，通过利用上下文适配器结构，我们使用时间与空间维度的形式化表示将不同任务概念注入模型，使其能够灵活处理任意视频合成任务。大量实验表明，VACE的统一模型在多个子任务上取得了与任务专用模型相当的性能。同时，它通过灵活的任务组合支持多样化应用。项目页面：https://ali-vilab.github.io/VACE-Page/。
英文摘要：
Diffusion Transformer has demonstrated powerful capability and scalability in generating high-quality images and videos. Further pursuing the unification of generation and editing tasks has yielded significant progress in the domain of image content creation. However, due to the intrinsic demands for consistency across both temporal and spatial dynamics, achieving a unified approach for video synthesis remains challenging. We introduce VACE, which enables users to perform Video tasks within an All-in-one framework for Creation and Editing. These tasks include reference-to-video generation, video-to-video editing, and masked video-to-video editing. Specifically, we effectively integrate the requirements of various tasks by organizing video task inputs, such as editing, reference, and masking, into a unified interface referred to as the Video Condition Unit (VCU). Furthermore, by utilizing a Context Adapter structure, we inject different task concepts into the model using formalized representations of temporal and spatial dimensions, allowing it to handle arbitrary video synthesis tasks flexibly. Extensive experiments demonstrate that the unified model of VACE achieves performance on par with task-specific models across various subtasks. Simultaneously, it enables diverse applications through versatile task combinations. Project page: https://ali-vilab.github.io/VACE-Page/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
