arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.03568cs.AI
收藏

The Matrix:具有实时移动控制的无限视界世界生成

The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control

Tongyi Lab(通义实验室);University of Waterloo(滑铁卢大学);University of Hong Kong(香港大学)

Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, Hongyang Zhang

AI总结:

本文提出 The Matrix 世界模拟器,融合 AAA 游戏监督数据与真实无监督影像,实现 720p、16 FPS、可实时控制的无限视界视频生成,并展现跨虚拟到现实的零样本泛化能力。

摘要:

我们提出了 The Matrix,这是首个基础级真实世界模拟器,能够生成连续的 720p 高保真实景视频流,并在第一人称和第三人称视角下提供实时、响应式控制,从而支持对高度动态环境进行沉浸式探索。The Matrix 使用来自《极限竞速:地平线 5》和《赛博朋克 2077》等 AAA 游戏的有限监督数据进行训练,并辅以东京街道等真实场景的大规模无监督影像,使用户能够在连续、未剪辑、长达一小时的序列中穿越沙漠、草原、水体和城市景观等多样地形。该系统以 16 FPS 运行,支持实时交互,并展示出零样本泛化能力,可将虚拟游戏环境迁移到通常难以收集连续移动数据的真实环境。例如,The Matrix 能够模拟一辆 BMW X3 驶过办公环境——这一场景既未出现在游戏数据中,也未出现在真实世界来源中。该方法展示了 AAA 游戏数据在推进鲁棒世界模型方面的潜力,并在数据有限的场景中弥合模拟与真实应用之间的差距。

英文摘要:

We present The Matrix, the first foundational realistic world simulator capable of generating continuous 720p high-fidelity real-scene video streams with real-time, responsive control in both first- and third-person perspectives, enabling immersive exploration of richly dynamic environments. Trained on limited supervised data from AAA games like Forza Horizon 5 and Cyberpunk 2077, complemented by large-scale unsupervised footage from real-world settings like Tokyo streets, The Matrix allows users to traverse diverse terrains -- deserts, grasslands, water bodies, and urban landscapes -- in continuous, uncut hour-long sequences. Operating at 16 FPS, the system supports real-time interactivity and demonstrates zero-shot generalization, translating virtual game environments to real-world contexts where collecting continuous movement data is often infeasible. For example, The Matrix can simulate a BMW X3 driving through an office setting--an environment present in neither gaming data nor real-world sources. This approach showcases the potential of AAA game data to advance robust world models, bridging the gap between simulations and real-world applications in scenarios with limited data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑