WorldSimBench: Towards Video Generation Models as World Simulators
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Beihang University(北京航空航天大学);The University of Hong Kong(香港大学)
Recent advancements in predictive models have demonstrated exceptional capabilities in predicting the future state of objects and scenes. However, the lack of categorization based on inherent characteristics continues to hinder the progress of predictive model development. Additionally, existing benchmarks are unable to effectively evaluate higher-capability, highly embodied predictive models from an embodied perspective. In this work, we classify the functionalities of predictive models into a hierarchy and take the first step in evaluating World Simulators by proposing a dual evaluation framework called WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, encompassing human preference assessments from the visual perspective and action-level evaluations in embodied tasks, covering three representative embodied scenarios: Open-Ended Embodied Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment dataset based on fine-grained human feedback, which we use to train a Human Preference Evaluator that aligns with human perception and explicitly assesses the visual fidelity of World Simulators. In the Implicit Manipulative Evaluation, we assess the video-action consistency of World Simulators by evaluating whether the generated situation-aware video can be accurately translated into the correct control signals in dynamic environments. Our comprehensive evaluation offers key insights that can drive further innovation in video generation models, positioning World Simulators as a pivotal advancement toward embodied artificial intelligence.
Endora: Video Generation Models as Endoscopy Simulators
The Chinese University of Hong Kong(香港中文大学);Massachusetts Institute of Technology(麻省理工学院);Centre for Artificial Intelligence and Robotics(人工智能与机器人中心);Shanghai AI Lab(上海人工智能实验室)
Max Planck Institute for Informatics(马克斯·普朗克信息研究所);Singapore University of Technology and Design(新加坡科技设计大学);A*STAR(新加坡科技研究局);Google(谷歌);Saarbrücken Research Center for Visual Computing, Interaction and Artificial Intelligence(斯图加特视觉计算、交互与人工智能研究中心)
Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation
Shanghai Jiao Tong University(上海交通大学);OpenGVLab(开放通用视觉实验室);Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);The University of Hong Kong(香港大学)
MeVGAN: GAN-based Plugin Model for Video Generation with Applications in Colonoscopy
Jagiellonian University(雅盖隆大学);Marian Smoluchowski Institute of Physics, Jagiellonian University(玛丽安·斯莫卢霍夫斯基物理研究所,雅盖隆大学);Skopia Medical Center(斯科皮亚医疗中心)
Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms
School of Computing and Data Science, The University of Hong Kong(计算与数据科学学院,香港大学);Hong Kong Generative AI Research and Development Center(香港生成式人工智能研究与开发中心)
DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenes
Zhejiang University(浙江大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Technical University of Munich(慕尼黑技术大学)
WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors
Tsinghua University(清华大学);Nanyang Technological University(南洋理工大学);University of Waterloo(滑铁卢大学);Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州))
ARDuP: Active Region Video Diffusion for Universal Policies
University of Maryland, College Park(马里兰大学帕克分校);University of Texas at Austin(德克萨斯大学奥斯汀分校);California Institute of Technology(加州理工学院);NVIDIA Corporation(英伟达公司)
DRAW2ACT: Turning Depth-Encoded Trajectories into Robotic Demonstration Videos
Ludwig Maximilian University of Munich(慕尼黑路德维希-马克西米利安大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);University of Freiburg(弗赖堡大学);Technical University of Munich(慕尼黑技术大学);Huawei Heisenberg Research Center (Munich)(华为海森堡研究中心(慕尼黑))