arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.18072cs.CV
收藏

WorldSimBench:迈向作为世界模拟器的视频生成模型

WorldSimBench: Towards Video Generation Models as World Simulators

The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Beihang University(北京航空航天大学);The University of Hong Kong(香港大学)

Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, Zhenfei Yin, Xihui Liu, Lu Sheng, Jing Shao, Lei Bai, Wanli Ouyang, Ruimao Zhang

AI总结:

本文提出双重评估框架 WorldSimBench,通过显式感知评估和隐式操纵评估,从视觉保真度与视频-动作一致性两个维度系统评估作为世界模拟器的视频生成模型,推动具身人工智能发展。

摘要:

近年来,预测模型在预测物体和场景未来状态方面展现出卓越能力。然而,缺乏基于内在特性的分类体系持续阻碍着预测模型的发展。此外,现有基准无法从具身视角有效评估能力更强、具身化程度更高的预测模型。在本工作中,我们将预测模型的功能划分为一个层级体系,并通过提出名为 WorldSimBench 的双重评估框架,迈出评估世界模拟器的第一步。WorldSimBench 包含显式感知评估和隐式操纵评估,涵盖从视觉视角进行的人类偏好评估以及具身任务中的动作级评估,覆盖三个代表性具身场景:开放式具身环境、自动驾驶和机器人操作。在显式感知评估中,我们引入了 HF-Embodied 数据集,这是一个基于细粒度人类反馈的视频评估数据集,我们利用该数据集训练了一个与人类感知对齐的人类偏好评估器,用于显式评估世界模拟器的视觉保真度。在隐式操纵评估中,我们通过评估生成的情境感知视频能否在动态环境中被准确转换为正确的控制信号,来评估世界模拟器的视频-动作一致性。我们的综合评估提供了关键见解,能够推动视频生成模型的进一步创新,将世界模拟器定位为迈向具身人工智能的关键进展。

英文摘要:

Recent advancements in predictive models have demonstrated exceptional capabilities in predicting the future state of objects and scenes. However, the lack of categorization based on inherent characteristics continues to hinder the progress of predictive model development. Additionally, existing benchmarks are unable to effectively evaluate higher-capability, highly embodied predictive models from an embodied perspective. In this work, we classify the functionalities of predictive models into a hierarchy and take the first step in evaluating World Simulators by proposing a dual evaluation framework called WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, encompassing human preference assessments from the visual perspective and action-level evaluations in embodied tasks, covering three representative embodied scenarios: Open-Ended Embodied Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment dataset based on fine-grained human feedback, which we use to train a Human Preference Evaluator that aligns with human perception and explicitly assesses the visual fidelity of World Simulators. In the Implicit Manipulative Evaluation, we assess the video-action consistency of World Simulators by evaluating whether the generated situation-aware video can be accurately translated into the correct control signals in dynamic environments. Our comprehensive evaluation offers key insights that can drive further innovation in video generation models, positioning World Simulators as a pivotal advancement toward embodied artificial intelligence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑