X-Foresight:一种通过预测世界建模的联合视觉-动作因果预测网络
X-Foresight: A Joint Vision-Action Causal Forecasting Network via Predictive World Modeling
AI总结:
提出X-Foresight,一种将预测世界模型直接集成到VLA架构中的方法,通过长程分块自回归策略和课程学习,联合学习世界建模与实时动作控制,以解决视频预测中的低熵冗余和长程因果建模难题。
摘要:
物理世界知识主要存在于视频中。赋予视觉-语言-动作(VLA)模型此类知识对于安全且可泛化的规划至关重要。预测世界建模通过从过去观测预测未来视频,使VLA能够内化物理动态和长程因果关系。然而,朴素的下一帧预测面临两个挑战:1)与语义上不同的文本标记不同,视频标记是低熵且冗余的,导致预测退化为琐碎的外推;2)世界建模存在时间困境:密集预测捕捉瞬时动态,但无法高效建模长程因果。为有效学习世界知识,我们引入X-Foresight,一种直接集成到VLA架构中的预测世界模型,以联合学习世界建模和实时动作控制。其核心是一种长程分块自回归策略,该策略解决了上述两个挑战:通过预测语义上遥远的块而非相邻帧,它避免了琐碎的外推,同时保留密集的块内帧用于瞬时动态和稀疏的块间过渡用于长程因果。课程学习计划逐步扩展预测范围并稳定长程训练。为有效捕捉长程因果,我们提出时间重要性采样,将监督集中于由自我运动和行为信号识别的安全关键块。我们进一步将逼真合成委托给基于扩散的多视图渲染器,以改善逼真外观。大量实验表明,X-Foresight在规划性能上显著优于VLA基线,同时保持强大的生成保真度,为世界知识驱动的自主系统建立了稳健的范式。
英文摘要:
Physical world knowledge resides mainly in videos. Equipping Vision-Language-Action (VLA) models with such knowledge is fundamental for safe and generalizable planning. Predictive world modeling enables VLA to internalize physical dynamics and long-term causality by predicting future video from past observations. However, naive next-frame prediction faces two challenges: 1) unlike semantically distinct text tokens, video tokens are low-entropy and redundant, causing prediction to degenerate into trivial extrapolation. 2) world modeling poses a temporal dilemma: dense prediction captures instantaneous dynamics, but cannot efficiently model long-horizon causality. To learn world knowledge effectively, we introduce X-Foresight, a predictive world model integrated directly into the VLA architecture to jointly learn world modeling and real-time action control. At its core lies a long-horizon chunk-wise auto-regressive strategy that addresses both challenges: by predicting semantically distant chunks rather than adjacent frames, it escapes trivial extrapolation, while preserving dense intra-chunk frames for instantaneous dynamics and sparse inter-chunk transitions for long-term causality. A curriculum learning schedule progressively extends prediction horizons and stabilizes long-horizon training. To capture long-term causality effectively, we present temporal importance sampling, which concentrates supervision on safety-critical chunks identified by ego-motion and behavioral signals. We further delegate photorealistic synthesis to a diffusion-based multi-view renderer, improving photorealistic appearance. Comprehensive experiments demonstrate that X-Foresight significantly outperforms VLA baselines in planning performance while maintaining strong generative fidelity, establishing a robust paradigm for world-knowledge-driven autonomous systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。