EndoGSim: 基于多模态大语言模型的物理感知4D动态内窥镜场景模拟
EndoGSim: Physics-Aware 4D Dynamic Endoscopic Scene Simulations via MLLM-Guided Gaussian Splatting
AI总结：
本文提出EndoGSim框架，通过MLLM引导的高斯点散布实现内窥镜场景的物理感知重建与模拟，结合预训练分割和深度估计，提升手术模拟的真实性和准确性。
摘要：
在机器人辅助微创手术中，高保真的动态内窥镜场景重建与模拟对于提升后续任务和改善手术结果至关重要。然而，现有方法主要关注视觉重建，缺乏用于真实模拟所需的物理描述。我们提出一个统一框架，通过多模态大语言模型（MLLM）引导的高斯点散布实现内窥镜场景的物理感知重建与模拟。我们的方法利用4D高斯点散布（4DGS）结合预训练分割和深度估计来表示可变形组织和工具。为了实现物理属性的自动推断，我们引入了物体级材料场，通过MLLM初始化材料参数并通过可微分材料点方法（MPM）进行细化，在渲染图像和光流的联合监督下进行。在开源和自建数据集上验证，我们的框架在模拟保真度和物理准确性方面优于现有方法，凸显其在机器人辅助手术应用中的潜力。
英文摘要：
In robot-assisted minimally invasive surgery, high-fidelity dynamic endoscopic scene reconstruction and simulation are crucial to enhancing downstream tasks and advancing surgical outcomes. However, existing methods primarily focus on visual reconstruction, lacking physics-based descriptions of the scene required for realistic simulation. We propose a unified framework that achieves physics-aware reconstruction and physical simulation of endoscopic scenes through Multi-modal Large Language Models (MLLMs)-guided Gaussian Splatting. Our approach utilizes 4D Gaussian Splatting (4DGS) integrated with pre-trained segmentation and depth estimation to represent deformable tissues and tools. To achieve automatic inference of physical properties, we introduce an object-wise material field that initializes material parameters via MLLM and refines them through a differentiable Material Point Method (MPM) under joint supervision from rendered images and optical flow. Validated on both open-source and in-house datasets, our framework achieves superior simulation fidelity and physical accuracy compared to state-of-the-art methods, underscoring its potential to advance robot-assisted surgical applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
