arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.07944cs.CVcs.AI
收藏

WorldGPT:一种受Sora启发的视频AI智能体,作为从文本和图像输入构建的丰富世界模型

WorldGPT: A Sora-Inspired Video AI Agent as Rich World Models from Text and Image Inputs

Harvard University(哈佛大学);Stanford University(斯坦福大学);Peking University(北京大学);University of Washington(华盛顿大学);Seeking AI

Deshun Yang, Luhui Hu, Yu Tian, Zihao Li, Chris Kelly, Bang Yang, Cindy Yang, Yuexian Zou

AI总结:

本文提出WorldGPT,一种受Sora启发的视频AI智能体,通过提示增强器和全视频翻译两部分,利用文本和图像输入构建世界模型,生成时间一致且动作平滑的视频,实验证明其有效性和新颖性。

摘要:

几种文本到视频的扩散模型在合成高质量视频内容方面已展现出令人称赞的能力。然而,在生成的序列中保持时间一致性和确保动作平滑性仍然是一个艰巨的挑战。在本文中,我们提出了一种创新的视频生成AI智能体,它利用受Sora启发的多模态学习的力量,基于文本提示和伴随图像构建熟练的世界模型框架。该框架包括两个部分:提示增强器和全视频翻译。第一部分利用ChatGPT的能力,为每个后续步骤精心提炼并主动构建精确的提示,从而保证提示传达的最大准确性以及后续模型操作中的精确执行。第二部分采用与现有先进扩散技术兼容的方法,在视频结尾处扩展生成并优化关键帧。然后,我们可以熟练地利用首尾关键帧的力量来制作具有增强时间一致性和动作平滑性的视频。实验结果证实,我们的方法在从文本和图像输入构建世界模型方面,比其他方法具有更强的有效性和新颖性。

英文摘要:

Several text-to-video diffusion models have demonstrated commendable capabilities in synthesizing high-quality video content. However, it remains a formidable challenge pertaining to maintaining temporal consistency and ensuring action smoothness throughout the generated sequences. In this paper, we present an innovative video generation AI agent that harnesses the power of Sora-inspired multimodal learning to build skilled world models framework based on textual prompts and accompanying images. The framework includes two parts: prompt enhancer and full video translation. The first part employs the capabilities of ChatGPT to meticulously distill and proactively construct precise prompts for each subsequent step, thereby guaranteeing the utmost accuracy in prompt communication and accurate execution in following model operations. The second part employ compatible with existing advanced diffusion techniques to expansively generate and refine the key frame at the conclusion of a video. Then we can expertly harness the power of leading and trailing key frames to craft videos with enhanced temporal consistency and action smoothness. The experimental results confirm that our method has strong effectiveness and novelty in constructing world models from text and image inputs over the other methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑