arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.27014cs.LG
收藏

基于JEPA的世界模型的泛化理论

A Generalization Theory for JEPA-Based World Models

State Key Lab of General AI, School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院通用人工智能国家重点实验室);University of Sydney(悉尼大学)

Jingyi Cui, Qi Zhang, Hongwei Wen, Letian Cheng, Yisen Wang

AI总结:

本文首次为JEPA世界模型建立泛化理论,将其预训练表述为条件谱图学习问题,并证明JEPA目标等价于动作条件共现矩阵的低秩分解,进而推导出有限样本泛化界,揭示了潜在维度在近似误差与样本误差之间的权衡。

摘要:

联合嵌入预测架构(JEPAs)最近作为一种有前景的世界模型范式出现,通过在潜在空间中学习预测动态,而不是在输入级别生成未来观测。尽管它们在经验上取得了成功,但对基于JEPA的世界模型的理论理解仍然有限。在本文中,我们为基于JEPA的世界模型建立了第一个泛化理论。我们将JEPA预训练表述为一个条件谱图学习问题,并证明JEPA目标等价于一个动作条件共现矩阵的低秩分解。基于这一特征,我们建立了JEPA预训练误差与下游规划遗憾之间的联系,从而得出了基于JEPA的世界模型的有限样本泛化界。我们的分析揭示了潜在维度上近似误差与样本误差之间的内在权衡,为潜在预测模型与输入级预测方法相比的优势和局限性提供了理论见解。

英文摘要:

Joint Embedding Predictive Architectures (JEPAs) have emerged as a promising paradigm for world modeling by predicting future representations rather than reconstructing observations. Despite their empirical success, the theoretical understanding of JEPA-based world models remains limited. In this paper, we develop the first generalization theory for JEPA-based world models. We formulate JEPA pretraining as a conditional spectral graph learning problem and show that the JEPA objective is equivalent to a low-rank factorization of an action-conditioned co-occurrence matrix. Building on this characterization, we establish a connection between JEPA pretraining error and downstream planning regret, leading to a finite-sample generalization bound for JEPA-based world models. Our analysis reveals an approximation-sample error trade-off governed by latent dimensionality, providing insights into latent- versus input-level prediction. The theoretical trade-off motivates us to investigate a simple Multi-Depth JEPA extension that combines representations from different network depths. In validation experiments, latent-level prediction exhibits greater robustness under unpredictable observations and difficult planning tasks. Multi-depth representations further improve planning success over single-depth representations and a dimensionality-matched control.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑