arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.09877cs.CVcs.AIcs.RO
收藏

Genie 4D:语义先验引导的4D动态场景重建

Genie 4D: Semantic-Prior-Guided 4D Dynamic Scene Reconstruction

University of Zurich(苏黎世大学);ETH Zurich(苏黎世联邦理工学院)

Yiru Yang, Zhuojie Wu, Nishant Kumar Singh, Max Schulthess

AI总结:

提出Genie 4D框架,结合实时视觉惯性高斯泼溅前端和前馈4D骨干网络,利用冻结的DINOv3特征作为结构先验抑制身份漂移,并通过条件扩散精炼器恢复高频细节,最终通过轻量级潜在动作头实现用户可控的4D世界模型重建。

摘要:

在计算机视觉与机器人感知的交汇处,动态场景的4D重建将低层几何感知与高层语义理解联系起来。我们提出Genie 4D,一个将手持手机拍摄转化为语义化、动作可控的4D世界模型的框架。Genie 4D将用于度量几何的实时视觉惯性高斯泼溅前端与由冻结的DINOv3特征(作为结构先验)正则化的前馈4D骨干网络相结合。语义先验抑制了动态跟踪中的身份漂移,而短条件扩散精炼器恢复了回归骨干网络平滑掉的高频表面细节。最后,一个轻量级潜在动作头将重建的4D状态暴露给以JEPA风格下一嵌入目标训练的Genie式世界模型,使得场景可以在用户动作下向前推进。在Point Odyssey和TUM-Dynamics基准测试上,Genie 4D保留了前馈基线的线性时间复杂度O(T),同时提高了3D跟踪精度(APD)和重建完整性,并且可以在单个消费级GPU(RTX 5090)上通过iPhone、Mac、Windows和Linux采集客户端交互式运行。Genie 4D为走向物理基础的世界模型提供了一条实用的、语义先验引导的路径。

英文摘要:

At the intersection of computer vision and robotic perception, 4D reconstruction of dynamic scenes connects low-level geometric sensing with high-level semantic understanding. We present Genie 4D, a framework that turns hand-held phone capture into a semantically grounded, action-controllable 4D world model. Genie 4D couples a real-time visual-inertial Gaussian splatting front-end for metric geometry with a feed-forward 4D backbone regularized by frozen DINOv3 features acting as structural priors. The semantic priors suppress identity drift during dynamic tracking, while a short conditional diffusion refiner recovers high-frequency surface detail that regression backbones smooth away. Finally, a lightweight latent-action head exposes the reconstructed 4D state to a Genie-style world model trained with a JEPA-style next-embedding objective, so that the scene can be rolled forward under user actions. On the Point Odyssey and TUM-Dynamics benchmarks, Genie 4D retains the linear time complexity O(T) of feed-forward baselines while improving 3D tracking accuracy (APD) and reconstruction completeness, and it runs interactively on a single consumer GPU (RTX 5090) from iPhone, Mac, Windows, and Linux capture clients. Genie 4D offers a practical, semantic-prior-guided path toward physically grounded world models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑