视觉补丁世界：作为规划潜在结构化表示的代码世界模型

中文摘要 AI 辅助

不同研究方向对世界模型的使用方式各异，但都旨在以支持感知、模拟和规划的形式捕捉世界在行动下的演变。两种主要实现方式是在连续向量空间中学习动力学的神经预测器和揭示显式状态与物理定律的手工物理引擎。神经预测器可从数据扩展但动力学形式隐含，物理引擎可检查和编辑但难以大规模构建。我们引入视觉补丁世界（VPW），它将世界动力学表示为代码。VPW首先通过短时间的主动探测选择定性动力学形式，然后通过最小化多步预测误差从记录的状态 - 动作轨迹中拟合该形式的自由参数。生成的程序可像模拟器一样向前推进、以源形式检查并用于模型预测控制；图像派生的场景图可在重新规划时提供实时状态。与基于代码的先前世界模型相比，VPW平均规划成功率达到69.0%，比最强的代码基线高出23.5分。在选择正确的定性动力学至关重要时收益最大。在相同规划器下，诱导模型在导航和丰富抓取控制方面接近真实引擎的成功率；在丰富接触推动方面仍有差距，在引擎中检查有前途的计划短名单可缩小大部分差距。这些结果为自动构建对规划有用的代码世界模型建立了实用途径。代码可在该https网址获取。

英文摘要

Different research lines use the term world model in different ways, yet they share a common aim: to capture how the world evolves under action in a form that supports perception, simulation, and planning. Two prominent realizations are neural predictors that learn dynamics in continuous vector spaces, and hand-built physics engines that expose explicit state and physical laws. Neural predictors scale from data but leave the form of the dynamics implicit; physics engines are inspectable and editable but difficult to construct at scale. We introduce VisualPatchWorld (VPW), which represents world dynamics as code. VPW first selects a qualitative dynamical form with short active probes, then fits that form's free parameters from recorded state-action traces by minimizing multi-step prediction error. The resulting programs can be rolled forward like a simulator, inspected in source form, and used inside model-predictive control; image-derived scene graphs can supply the live state at replan time. Across comparisons with prior code-based world models, VPW attains 69.0% mean planning success and exceeds the strongest code baseline by 23.5 points. The largest gains arise when choosing the correct qualitative dynamics is essential. Under the same planner, the induced models approach ground-truth engine success on navigation and grasp-rich control; a residual gap remains for contact-rich pushing, and checking a shortlist of promising plans in the engine closes most of that gap. These results establish a practical route toward automatically constructed code world models that are useful for planning. Code is available at https://github.com/HKBU-KnowComp/VisualPatchWorld/.

发表机构

• Hong Kong Baptist University（香港浸会大学）

机构由 AI 辅助整理，请以论文原文为准。