arXiv 2411.049252024-11-11cs.CVcs.AIcs.MA

StoryAgent：基于多智能体协作的定制化叙事视频生成

StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration

Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；United Arab Emirates University（阿拉伯联合酋长国大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）

Panwen Hu, Jin Jiang, Jianqi Chen, Mingfei Han, Shengcai Liao, Xiaojun Chang, Xiaodan Liang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

StoryAgent提出多智能体协作框架，分解定制化叙事视频生成任务，结合LoRA-BE等方法提升角色与时间一致性，优于现有技术。

摘要：

人工智能生成内容（AIGC）的出现推动了自动化视频生成的研究，以简化传统流程。然而，自动化叙事视频制作，特别是针对定制化叙事的制作，由于在镜头间保持主体一致性的复杂性，仍然具有挑战性。尽管现有方法如Mora和AesopAgent集成了多个智能体用于故事到视频（S2V）生成，但它们在保持主角一致性以及支持定制化叙事视频生成（CSVG）方面存在不足。为解决这些局限性，我们提出了StoryAgent，一个专为CSVG设计的多智能体框架。StoryAgent将CSVG分解为分配给专门智能体的不同子任务，模拟专业制作流程。值得注意的是，我们的框架包括故事设计、分镜生成、视频创作、智能体协调和结果评估等智能体。利用不同模型的优势，StoryAgent增强了对生成过程的控制，显著提高了角色一致性。具体而言，我们引入了一种定制化的图像到视频（I2V）方法LoRA-BE，以增强镜头内的时间一致性，同时提出了一种新颖的分镜生成流程，以保持镜头间的主体一致性。大量实验证明了我们的方法在合成高度一致的叙事视频方面的有效性，优于最先进的方法。我们的贡献包括引入StoryAgent，一个用于视频生成任务的多功能框架，以及用于保持主角一致性的新技术。

英文摘要：

The advent of AI-Generated Content (AIGC) has spurred research into automated video generation to streamline conventional processes. However, automating storytelling video production, particularly for customized narratives, remains challenging due to the complexity of maintaining subject consistency across shots. While existing approaches like Mora and AesopAgent integrate multiple agents for Story-to-Video (S2V) generation, they fall short in preserving protagonist consistency and supporting Customized Storytelling Video Generation (CSVG). To address these limitations, we propose StoryAgent, a multi-agent framework designed for CSVG. StoryAgent decomposes CSVG into distinct subtasks assigned to specialized agents, mirroring the professional production process. Notably, our framework includes agents for story design, storyboard generation, video creation, agent coordination, and result evaluation. Leveraging the strengths of different models, StoryAgent enhances control over the generation process, significantly improving character consistency. Specifically, we introduce a customized Image-to-Video (I2V) method, LoRA-BE, to enhance intra-shot temporal consistency, while a novel storyboard generation pipeline is proposed to maintain subject consistency across shots. Extensive experiments demonstrate the effectiveness of our approach in synthesizing highly consistent storytelling videos, outperforming state-of-the-art methods. Our contributions include the introduction of StoryAgent, a versatile framework for video generation tasks, and novel techniques for preserving protagonist consistency.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。