arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.13248cs.CV
收藏

Mora:通过多代理框架实现通用视频生成

Mora: Enabling Generalist Video Generation via A Multi-Agent Framework

University of Notre Dame(圣母大学);Lehigh University(理海大学);LinkedIn Corporation(领英公司);MBZUAI(穆罕默德·本·扎耶德人工智能大学);Nanyang Technological University(南洋理工大学);Peking University(北京大学);Microsoft Research(微软研究院)

Zhengqing Yuan, Yixin Liu, Yihan Cao, Weixiang Sun, Haolong Jia, Ruoxi Chen, Zhaoxu Li, Bin Lin, Li Yuan, Lifang He, Chi Wang, Yanfang Ye, Lichao Sun

摘要:

文本到视频的生成已经取得了重大进展,但由于其闭源性质,复制OpenAI Sora等先进系统的功能仍然具有挑战性。现有的开源方法很难达到可比的性能,通常受到无效的代理协作和不足的训练数据质量的阻碍。在本文中,我们介绍莫拉,一种新的多代理框架,利用现有的开源模块复制Sora功能。我们通过提出三个关键技术来解决这些基本限制:(1)具有自调节因子的多智能体微调,以增强智能体间的协调,(2)使用大型模型来合成训练数据的无数据训练策略,以及(3)与多模态大语言模型相结合的人在回路机制,用于数据过滤,以确保高质量的训练数据集。我们对六个视频生成任务的综合实验表明,Mora在VBench上实现了与Sora相当的性能,在各种任务中优于现有的开源方法。具体来说,在文本到视频生成任务中,Mora的视频质量得分为0.800,超过了Sora 0.797,并在六个关键指标上优于所有其他基准模型。此外,在图像到视频生成任务中,Mora获得了1.00的完美动态度分数,证明了在增强运动真实感和实现比Sora更高的成像质量方面的卓越能力。这些结果突出了协作多智能体系统和人在循环机制在推进文本到视频生成的潜力。我们的代码可在\url{https://github.com/lichao-sun/Mora}上获得。

英文摘要:

Text-to-video generation has made significant strides, but replicating the capabilities of advanced systems like OpenAI Sora remains challenging due to their closed-source nature. Existing open-source methods struggle to achieve comparable performance, often hindered by ineffective agent collaboration and inadequate training data quality. In this paper, we introduce Mora, a novel multi-agent framework that leverages existing open-source modules to replicate Sora functionalities. We address these fundamental limitations by proposing three key techniques: (1) multi-agent fine-tuning with a self-modulation factor to enhance inter-agent coordination, (2) a data-free training strategy that uses large models to synthesize training data, and (3) a human-in-the-loop mechanism combined with multimodal large language models for data filtering to ensure high-quality training datasets. Our comprehensive experiments on six video generation tasks demonstrate that Mora achieves performance comparable to Sora on VBench, outperforming existing open-source methods across various tasks. Specifically, in the text-to-video generation task, Mora achieved a Video Quality score of 0.800, surpassing Sora 0.797 and outperforming all other baseline models across six key metrics. Additionally, in the image-to-video generation task, Mora achieved a perfect Dynamic Degree score of 1.00, demonstrating exceptional capability in enhancing motion realism and achieving higher Imaging Quality than Sora. These results highlight the potential of collaborative multi-agent systems and human-in-the-loop mechanisms in advancing text-to-video generation. Our code is available at \url{https://github.com/lichao-sun/Mora}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑