RACCooN：一种具备自动生成叙事功能的通用教学视频编辑框架
RACCooN: A Versatile Instructional Video Editing Framework with Auto-Generated Narratives
AI总结：
针对现有视频生成模型依赖人工文本提示、灵活性不足的问题，本文提出RACCooN通用视频编辑框架，通过视频转段落、段落转视频两阶段及多粒度时空池化等技术，支持多种视频编辑操作，效果优异且可集成到其他最优模型中。
摘要：
当前的视频生成模型主要依赖精心撰写的文本提示完成特定任务，例如图像修复或风格编辑。这类模型需要为输入视频编写耗费大量人力的文本描述，限制了其将个人/原始视频调整至符合用户需求的灵活性。本文提出RACCooN，这是一个通用且用户友好的“视频-段落-视频”生成框架，通过统一流水线支持移除、添加、修改等多种视频编辑功能。RACCooN包含两个主要阶段：视频转段落（V2P）和段落转视频（P2V）。在V2P阶段，我们用结构清晰的自然语言自动描述视频场景，同时捕捉整体上下文和聚焦的对象细节。随后在P2V阶段，用户可选择性地优化这些描述，以引导视频扩散模型，实现对输入视频的多种修改，例如移除、更换主体和/或添加新对象。该方法相较于其他方法有几项显著贡献：（1）RACCooN提出了一种多粒度时空池化策略来生成结构清晰的视频描述，无需复杂的人工标注即可同时捕捉全局上下文和对象细节，简化了用户基于文本的精准视频内容编辑流程。（2）我们的视频生成模型融合了自动生成的叙事或指令，提升了生成内容的质量和准确性。（3）RACCooN还可规划在给定视频中构想新对象，用户只需向模型发出提示，就能获得用于复杂视频编辑的详细编辑方案。该框架在视频转段落生成、视频内容编辑方面展现出出色的通用能力，还可集成到其他SOTA（当前最优）视频生成模型中以实现进一步优化。
英文摘要：
Recent video generative models primarily rely on carefully written text prompts for specific tasks, like inpainting or style editing. They require labor-intensive textual descriptions for input videos, hindering their flexibility to adapt personal/raw videos to user specifications. This paper proposes RACCooN, a versatile and user-friendly video-to-paragraph-to-video generative framework that supports multiple video editing capabilities such as removal, addition, and modification, through a unified pipeline. RACCooN consists of two principal stages: Video-to-Paragraph (V2P) and Paragraph-to-Video (P2V). In the V2P stage, we automatically describe video scenes in well-structured natural language, capturing both the holistic context and focused object details. Subsequently, in the P2V stage, users can optionally refine these descriptions to guide the video diffusion model, enabling various modifications to the input video, such as removing, changing subjects, and/or adding new objects. The proposed approach stands out from other methods through several significant contributions: (1) RACCooN suggests a multi-granular spatiotemporal pooling strategy to generate well-structured video descriptions, capturing both the broad context and object details without requiring complex human annotations, simplifying precise video content editing based on text for users. (2) Our video generative model incorporates auto-generated narratives or instructions to enhance the quality and accuracy of the generated content. (3) RACCooN also plans to imagine new objects in a given video, so users simply prompt the model to receive a detailed video editing plan for complex video editing. The proposed framework demonstrates impressive versatile capabilities in video-to-paragraph generation, video content editing, and can be incorporated into other SoTA video generative models for further enhancement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
