arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.07952cs.CVcs.AIcs.MM
收藏

AesopAgent:故事到视频制作的智能体驱动进化系统

AesopAgent: Agent-driven Evolutionary System on Story-to-Video Production

DAMO Academy, Alibaba Group(阿里巴巴达摩院)

Jiuniu Wang, Zehua Du, Yuyuan Zhao, Bo Yuan, Kexiang Wang, Jian Liang, Yaxi Zhao, Yihen Lu, Gengliang Li, Junlong Gao, Xin Tu, Zhenyu Guo

AI总结:

AesopAgent提出智能体驱动的进化系统,通过水平层RAG优化工作流和工具层保证一致性,将故事自动生成连贯视频,达到先进水平。

摘要:

智能体(Agent)和AIGC(人工智能生成内容)技术近期取得了显著进展。我们提出了AesopAgent,一个用于故事到视频制作的智能体驱动进化系统。AesopAgent是智能体技术在多模态内容生成中的实际应用。该系统将多种生成能力集成在一个统一框架内,使个人用户能够轻松利用这些模块。这一创新系统将用户的故事提案转换为脚本、图像和音频,然后将这些多模态内容整合成视频。此外,动画单元(如Gen-2和Sora)可以使视频更具感染力。AesopAgent系统能够编排视频生成的任务工作流,确保生成的视频内容丰富且连贯。该系统主要包含两层,即水平层(Horizontal Layer)和工具层(Utility Layer)。在水平层中,我们引入了一种新颖的基于RAG(检索增强生成)的进化系统,用于优化整个视频生成工作流及工作流中的各个步骤。它通过积累专家经验和专业知识,持续进化并迭代优化工作流,包括优化LLM提示词和工具的使用。工具层提供多种工具,实现图像生成的一致性,使图像在构图、角色和风格上视觉连贯。同时,它提供音频和特效,并将其整合为富有表现力且逻辑编排的视频。总体而言,我们的AesopAgent在视觉叙事方面相较于许多先前工作达到了最先进的性能。AesopAgent旨在为个人用户提供便捷服务,可在以下页面获取:https://aesopai.github.io/。

英文摘要:

The Agent and AIGC (Artificial Intelligence Generated Content) technologies have recently made significant progress. We propose AesopAgent, an Agent-driven Evolutionary System on Story-to-Video Production. AesopAgent is a practical application of agent technology for multimodal content generation. The system integrates multiple generative capabilities within a unified framework, so that individual users can leverage these modules easily. This innovative system would convert user story proposals into scripts, images, and audio, and then integrate these multimodal contents into videos. Additionally, the animating units (e.g., Gen-2 and Sora) could make the videos more infectious. The AesopAgent system could orchestrate task workflow for video generation, ensuring that the generated video is both rich in content and coherent. This system mainly contains two layers, i.e., the Horizontal Layer and the Utility Layer. In the Horizontal Layer, we introduce a novel RAG-based evolutionary system that optimizes the whole video generation workflow and the steps within the workflow. It continuously evolves and iteratively optimizes workflow by accumulating expert experience and professional knowledge, including optimizing the LLM prompts and utilities usage. The Utility Layer provides multiple utilities, leading to consistent image generation that is visually coherent in terms of composition, characters, and style. Meanwhile, it provides audio and special effects, integrating them into expressive and logically arranged videos. Overall, our AesopAgent achieves state-of-the-art performance compared with many previous works in visual storytelling. Our AesopAgent is designed for convenient service for individual users, which is available on the following page: https://aesopai.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑