VDAWorld: 通过VLM导向的抽象与模拟进行世界建模
VDAWorld: World Modelling via VLM-Directed Abstraction and Simulation
AI总结：
提出VDAWorld框架，利用视觉语言模型自主构建场景表示并选择物理模拟器，通过抽象与自适应模拟实现高质量世界建模，在交互控制、反事实生成和物理逻辑推理任务上取得最优结果。
摘要：
生成式视频模型作为世界建模的主要方法，面临根本性限制。它们常常违反物理和逻辑规则，缺乏交互性，并且作为不透明的黑箱，不适合构建结构化、可查询的世界。为了克服这些挑战，我们提出了一种新范式，专注于将图像-文本对提炼为一种易于处理的、适用于模拟的抽象表示。我们引入了VDAWorld，这是一个框架，其中视觉语言模型（VLM）作为智能代理来协调这一过程。VLM通过从一套视觉工具中选择，自主构建一个基于场景的（2D或3D）场景表示，并相应地选择兼容的物理模拟器（例如刚体、流体）对其施加作用。然后，VDAWorld可以从静态场景推断潜在动力学，以预测可能的未来状态。我们的实验表明，这种智能抽象与自适应模拟的结合产生了一个多功能的世界模型，能够在各种场景中生成高质量的模拟。我们展示了VDAWorld在交互控制、反事实生成以及物理和逻辑推理方面的多个应用，在多个基准测试中取得了最先进的结果。
英文摘要：
Generative video models, a leading approach to world modelling, face fundamental limitations. They often violate physical and logical rules, lack interactivity, and operate as opaque black boxes ill-suited for building structured, queryable worlds. To overcome these challenges, we propose a new paradigm focused on distilling an image caption pair into a tractable, abstract representation optimized for simulation. We introduce VDAWorld, a framework where a Vision-Language Model (VLM) acts as an intelligent agent to orchestrate this process. The VLM autonomously constructs a grounded (2D or 3D) scene representation by selecting from a suite of vision tools, and accordingly chooses a compatible physics simulator (e.g., rigid body, fluid) to act upon it. VDAWorld can then infer latent dynamics from the static scene to predict plausible future states. Our experiments show that this combination of intelligent abstraction and adaptive simulation results in a versatile world model capable of producing high quality simulations across a wide range of scenarios. We demonstrate several applications of VDAWorld across interactive control and counterfactual generation, and physical and logical reasoning, achieving state-of-the-art results on several benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
