arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.16211cs.CVcs.CLcs.GR
收藏

你的世界模拟器是一个好的故事呈现器吗?面向未来长视频生成的连续事件基准

Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation

University of Washington(华盛顿大学);Georgia Institute of Technology(佐治亚理工学院);University of California, San Diego(加利福尼亚大学圣地亚哥分校);Microsoft(微软公司)

Yiping Wang, Xuehai He, Kuan Wang, Luyao Ma, Jianwei Yang, Shuohang Wang, Simon Shaolei Du, Yelong Shen

AI总结:

引入StoryEval基准,通过423个连续事件提示评估T2V模型的故事完成能力,揭示现有模型在连贯故事呈现上的不足。

摘要:

当前最先进的视频生成模型能够生成具有高度真实细节的商业级视频。然而,它们仍然难以连贯地呈现提示词所指定故事中的多个顺序事件,这可以预见是未来长视频生成场景的一项关键能力。例如,顶级T2V生成模型仍然无法生成关于“如何把大象放进冰箱”这一简短简单故事的视频。现有的注重细节的基准主要关注美学质量和时空一致性等细粒度指标,但在评估模型处理事件级故事呈现的能力方面存在不足。为了弥补这一差距,我们引入了StoryEval,这是一个面向故事的基准,专门设计用于评估文本到视频(T2V)模型的故事完成能力。StoryEval包含423个提示词,涵盖7个类别,每个提示词代表由2-4个连续事件组成的短故事。我们采用先进的视觉语言模型,如GPT-4V和LLaVA-OV-Chat-72B,来验证生成视频中每个事件的完成情况,并应用一致投票方法来增强可靠性。我们的方法确保与人类评估的高度一致性,对11个模型的评估揭示了其挑战性,没有模型超过50%的平均故事完成率。StoryEval为推进T2V模型提供了新的基准,并突出了开发下一代连贯故事驱动视频生成解决方案的挑战与机遇。

英文摘要:

The current state-of-the-art video generative models can produce commercial-grade videos with highly realistic details. However, they still struggle to coherently present multiple sequential events in the stories specified by the prompts, which is foreseeable an essential capability for future long video generation scenarios. For example, top T2V generative models still fail to generate a video of the short simple story 'how to put an elephant into a refrigerator.' While existing detail-oriented benchmarks primarily focus on fine-grained metrics like aesthetic quality and spatial-temporal consistency, they fall short of evaluating models' abilities to handle event-level story presentation. To address this gap, we introduce StoryEval, a story-oriented benchmark specifically designed to assess text-to-video (T2V) models' story-completion capabilities. StoryEval features 423 prompts spanning 7 classes, each representing short stories composed of 2-4 consecutive events. We employ advanced vision-language models, such as GPT-4V and LLaVA-OV-Chat-72B, to verify the completion of each event in the generated videos, applying a unanimous voting method to enhance reliability. Our methods ensure high alignment with human evaluations, and the evaluation of 11 models reveals its challenge, with none exceeding an average story-completion rate of 50%. StoryEval provides a new benchmark for advancing T2V models and highlights the challenges and opportunities in developing next-generation solutions for coherent story-driven video generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑