SeeU: 通过4D动态感知生成可见的世界
SeeU: Seeing the Unseen World via 4D Dynamics-aware Generation
AI总结：
SeeU通过4D动态感知生成框架，从稀疏单目2D帧重建4D世界，学习连续4D动态并生成未见视觉内容，实现物理一致的连续生成。
摘要：
图像和视频是4D世界（3D空间+时间）的离散2D投影。大多数视觉理解、预测和生成直接操作2D观察，导致性能不佳。我们提出SeeU，一种学习连续4D动态并生成未见视觉内容的新方法。SeeU的核心原理是一种新的2D→4D→2D学习框架。SeeU首先从稀疏和单目2D帧重建4D世界（2D→4D）。然后在低秩表示和物理约束上学习连续4D动态（离散4D→连续4D）。最后，SeeU向前推进时间，将世界重新投影回2D在采样时间和视角，并基于时空上下文生成未见区域（4D→2D）。通过在4D中建模动态，SeeU实现了连续且物理一致的新型视觉生成，展示了在多个任务中的强大潜力，包括未见时间生成、未见空间生成和视频编辑。所有数据和代码将在https://yuyuanspace.com/SeeU/公开。
英文摘要：
Images and videos are discrete 2D projections of the 4D world (3D space + time). Most visual understanding, prediction, and generation operate directly on 2D observations, leading to suboptimal performance. We propose SeeU, a novel approach that learns the continuous 4D dynamics and generate the unseen visual contents. The principle behind SeeU is a new 2D$\to$4D$\to$2D learning framework. SeeU first reconstructs the 4D world from sparse and monocular 2D frames (2D$\to$4D). It then learns the continuous 4D dynamics on a low-rank representation and physical constraints (discrete 4D$\to$continuous 4D). Finally, SeeU rolls the world forward in time, re-projects it back to 2D at sampled times and viewpoints, and generates unseen regions based on spatial-temporal context awareness (4D$\to$2D). By modeling dynamics in 4D, SeeU achieves continuous and physically-consistent novel visual generation, demonstrating strong potentials in multiple tasks including unseen temporal generation, unseen spatial generation, and video editing. All data and code will be public at https://yuyuanspace.com/SeeU/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
