arXiv 2512.231622026-03-11cs.ROcs.CV

Cosmos-H-Surgical: 通过世界建模学习手术机器人策略

Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling

NVIDIA；The Chinese University of Hong Kong（香港中文大学）；Sung Kyun Kwan University（全州大学）；Wenzhou Medical University（温州医学院）；National University of Singapore（新加坡国立大学）；Ruijin Hospital（瑞金医院）

Yufan He, Pengfei Guo, Mengya Xu, Zhaoshuo Li, Andriy Myronenko, Dillan Imans, Bingjie Liu, Dongren Yang, Mingxue Gu, Yongnan Ji, Yueming Jin, Ren Zhao, Baiyong Shen, Daguang Xu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出通过世界建模学习手术机器人策略，利用合成数据和逆动力学模型提升自主手术能力。

摘要：

数据稀缺仍然是实现完全自主手术机器人的根本障碍。尽管大规模视觉语言动作（VLA）模型通过利用来自不同领域的配对视频动作数据，在家庭和工业操作中展示了令人印象深刻的泛化能力，但手术机器人却面临缺乏包含视觉观测和准确机器人动力学的数据库的问题。相比之下，存在大量手术视频，但缺乏相应的动作标签，阻碍了直接应用模仿学习或VLA训练。在本工作中，我们旨在通过Cosmos-H-Surgical，一个为手术物理AI设计的世界模型，来缓解这一问题。我们编纂了详细的手术动作文本对齐（SATA）数据集，专门用于手术机器人。然后我们基于最先进的物理AI世界模型和SATA构建了Cosmos-H-Surgical。它能够生成多样化、可泛化且逼真的手术视频。我们也是首次使用逆动力学模型从合成手术视频中推断伪动力学，生成合成的配对视频动作数据。我们证明了使用这些增强数据训练的手术VLA策略在真实手术机器人平台上显著优于仅使用真实演示数据训练的模型。我们的方法通过利用大量未标记的手术视频和生成式世界建模，提供了一条可扩展的路径，以实现自主手术技能的获取，从而为通用且数据高效的手术机器人策略打开了大门。

英文摘要：

Data scarcity remains a fundamental barrier to achieving fully autonomous surgical robots. While large scale vision language action (VLA) models have shown impressive generalization in household and industrial manipulation by leveraging paired video action data from diverse domains, surgical robotics suffers from the paucity of datasets that include both visual observations and accurate robot kinematics. In contrast, vast corpora of surgical videos exist, but they lack corresponding action labels, preventing direct application of imitation learning or VLA training. In this work, we aim to alleviate this problem by learning policy models from Cosmos-H-Surgical, a world model designed for surgical physical AI. We curated the Surgical Action Text Alignment (SATA) dataset with detailed action description specifically for surgical robots. Then we built Cosmos-H-Surgical based on the most advanced physical AI world model and SATA. It's able to generate diverse, generalizable and realistic surgery videos. We are also the first to use an inverse dynamics model to infer pseudokinematics from synthetic surgical videos, producing synthetic paired video action data. We demonstrate that a surgical VLA policy trained with these augmented data significantly outperforms models trained only on real demonstrations on a real surgical robot platform. Our approach offers a scalable path toward autonomous surgical skill acquisition by leveraging the abundance of unlabeled surgical video and generative world modeling, thus opening the door to generalizable and data efficient surgical robot policies.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。