arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.30239cs.CV
收藏

SAM3D-Phys:迈向真实世界中的多物体交互仿真

CA-World: Multi-Object Counterfactual Alignment for Efficient Interactive-Ready Reconstruction

Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Pengcheng Laboratory(鹏城实验室)

Xin Dong, Weijian Deng, Lihan Zhang, Tianru Dai, Wenfeng Deng, Yansong Tang

AI总结:

提出SAM3D-Phys框架,结合场景重建与SAM3D生成式先验,从部分观测中恢复完整可仿真物体几何,并通过物理约束优化和掩码引导外观蒸馏实现场景一致性,支持多物体同时交互仿真。

摘要:

这项工作解决了从重建的真实世界场景中恢复完整、可仿真的物体几何的问题,使得与场景中嵌入的物体进行基于物理的交互成为可能。虽然现代多视图重建方法可以产生视觉上准确的环境,但由于遮挡和有限的观测,物体往往不完整,因此不适合物理仿真。为了解决这一局限性,我们提出了SAM3D-Phys,一个将场景重建与SAM3D的生成式3D先验相结合以恢复可物理仿真的物体的框架。我们的方法首先从多视图图像重建场景,获得场景几何和物体的部分观测。然后,我们利用SAM3D从这些部分观测中推断出完整的物体几何。为了确保恢复的物体与重建场景保持一致,我们通过两种互补策略恢复场景一致的物体状态:一种物理约束的空间优化算法,迭代地将恢复的物体对齐到其原始位置;以及一种掩码引导的外观蒸馏模块,基于观测图像细化纹理保真度。通过恢复完整的物体几何并在场景中恢复其姿态和外观,SAM3D-Phys产生了适用于基于物理仿真的干净物体表示,使得在重建场景中能够对多个物体进行同时且物理一致的交互仿真。项目页面:https://chnxindong.github.io/sam3d-phys/

英文摘要:

Reconstructing interaction-ready 3D worlds is essential for physical simulation, virtual reality, robotics, and autonomous driving. However, existing methods mainly optimize static and holistic visual fidelity, with limited support for multi-object interaction. We argue that an interaction-ready reconstruction should anticipate potential scene changes and preserve geometric completeness, visual quality, multi-object spatial relationship, and physical plausibility under potential interactions. To this end, motivated by the causal intervention, we propose CA-World, an efficient framework that integrates counterfactual alignment learning into a decoupling-reintegration reconstruction pipeline. Specifically, we formulate foreground-background decoupling as a visual intervention, separate object generation and background inpainting as counterfactual generation, and scene reintegration as an inverse intervention. According to counterfactual consistency, reversing the intervention should recover the factual world, motivating three alignment objectives between the reintegrated and original scenes: appearance, spatial, and physical consistency. This formulates interaction-ready reconstruction as counterfactual alignment learning with direct supervision. Moreover, leveraging the locality of object-level interventions, CA-World constrains counterfactual states using the observed scene, enabling efficient and coherent reintegration without jointly optimizing all object states, thereby reducing computational cost and error accumulation. Experiments on object completeness, spatial accuracy, outdoor background completion, rendering quality, simulated dynamics, and downstream applications demonstrate the effectiveness of CA-World. Project page: https://chnxindong.github.io/ca-world/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑