CRISP: 从单目视频中基于接触引导的现实到仿真转换
CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives
AI总结：
CRISP通过拟合平面原始体和强化学习，从单目视频中生成可模拟的现实到仿真环境，显著降低运动跟踪失败率。
摘要：
我们介绍了CRISP，一种从单目视频中恢复可模拟的人类运动和场景几何的方法。先前关于联合人类-场景重建的工作依赖于数据驱动的先验知识和联合优化，没有在环路中包含物理，或者恢复出带有伪影的几何结构，导致在场景交互中运动跟踪策略失效。相比之下，我们的关键见解是通过在深度、法线和流上进行简单的聚类流程，拟合平面原始体来恢复凸的、干净的、可模拟的几何结构。为了重建可能在交互过程中被遮挡的场景几何，我们利用人类-场景接触建模（例如，我们使用人类姿态来重建椅子的被遮挡座位）。最后，我们通过使用强化学习驱动的人形控制器，确保人类和场景重建在物理上是合理的。我们的方法将运动跟踪失败率从人类中心视频基准（EMDB、PROX）上的55.2%降低到6.9%，同时实现了43%更快的RL仿真吞吐量。我们进一步在真实世界视频上验证了该方法，包括随意拍摄的视频、网络视频，甚至Sora生成的视频。这证明了CRISP在大规模生成物理有效的运动和交互环境方面的能力，大大推进了机器人和AR/VR的现实到仿真的应用。
英文摘要：
We introduce CRISP, a method that recovers simulatable human motion and scene geometry from monocular video. Prior work on joint human-scene reconstruction relies on data-driven priors and joint optimization with no physics in the loop, or recovers noisy geometry with artifacts that cause motion tracking policies with scene interactions to fail. In contrast, our key insight is to recover convex, clean, and simulation-ready geometry by fitting planar primitives to a point cloud reconstruction of the scene, via a simple clustering pipeline over depth, normals, and flow. To reconstruct scene geometry that might be occluded during interactions, we make use of human-scene contact modeling (e.g., we use human posture to reconstruct the occluded seat of a chair). Finally, we ensure that human and scene reconstructions are physically-plausible by using them to drive a humanoid controller via reinforcement learning. Our approach reduces motion tracking failure rates from 55.2\% to 6.9\% on human-centric video benchmarks (EMDB, PROX), while delivering a 43\% faster RL simulation throughput. We further validate it on in-the-wild videos including casually-captured videos, Internet videos, and even Sora-generated videos. This demonstrates CRISP's ability to generate physically-valid human motion and interaction environments at scale, greatly advancing real-to-sim applications for robotics and AR/VR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
