arXiv 2509.231072025-10-27cs.ROcs.AI

开放词汇时空场景图用于机器人感知与遥操作规划

Open-Vocabulary Spatio-Temporal Scene Graph for Robot Perception and Teleoperation Planning

HI-Robot Lab, School of Automation Science and Engineering, South China University of Technology（华南理工大学自动化科学与工程学院）；Institute of AI Industries, Chinese Academy of Science（中国科学院人工智能产业研究所）；School of Computer Science, Central China Normal University（中央财经大学计算机科学学院）；School of Computer Science and Informatics, University of Liverpool（利物浦大学计算机科学与信息学院）

Yi Wang, Zeyu Xue, Mujie Liu, Tongqin Zhang, Yan Hu, Zhou Zhao, Chenguang Yang, Zhenyu Lu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出时空开放词汇场景图（ST-OVSG），结合时间动态与延迟标注，解决遥操作中传输延迟导致的命令误解，提升规划鲁棒性，在Replica基准上节点准确率达74%，延迟鲁棒实验中规划成功率达70.5%。

摘要：

通过自然语言进行遥操作可降低操作员的工作负担，并提高高风险或远程环境中的安全性。然而，在动态远程场景中，双向通信期间的传输延迟会在远程感知状态与操作员意图之间产生差距，导致命令误解和错误执行。为缓解这一问题，我们引入了时空开放词汇场景图（ST-OVSG），这是一种将开放词汇感知与时间动态和轻量级延迟标注相结合的表征。ST-OVSG利用大型视觉语言模型（LVLMs）构建开放词汇的3D对象表征，并通过带有我们提出的时间匹配代价的匈牙利匹配算法将其扩展到时间域，从而生成统一的时空场景图。其中嵌入了一个延迟标签，使LVLM规划器能够回顾性地查询过去的场景状态，从而解决由传输延迟引起的局部-远程状态不匹配问题。为进一步减少冗余并突出任务相关线索，我们提出了一种面向任务的子图过滤策略，为规划器生成紧凑的输入。ST-OVSG可泛化到新类别，并在无需微调的情况下增强规划对传输延迟的鲁棒性。实验表明，我们的方法在Replica基准上达到了74%的节点准确率，优于ConceptGraph。值得注意的是，在延迟鲁棒性实验中，由ST-OVSG辅助的LVLM规划器实现了70.5%的规划成功率。

英文摘要：

Teleoperation via natural-language reduces operator workload and enhances safety in high-risk or remote settings. However, in dynamic remote scenes, transmission latency during bidirectional communication creates gaps between remote perceived states and operator intent, leading to command misunderstanding and incorrect execution. To mitigate this, we introduce the Spatio-Temporal Open-Vocabulary Scene Graph (ST-OVSG), a representation that enriches open-vocabulary perception with temporal dynamics and lightweight latency annotations. ST-OVSG leverages LVLMs to construct open-vocabulary 3D object representations, and extends them into the temporal domain via Hungarian assignment with our temporal matching cost, yielding a unified spatio-temporal scene graph. A latency tag is embedded to enable LVLM planners to retrospectively query past scene states, thereby resolving local-remote state mismatches caused by transmission delays. To further reduce redundancy and highlight task-relevant cues, we propose a task-oriented subgraph filtering strategy that produces compact inputs for the planner. ST-OVSG generalizes to novel categories and enhances planning robustness against transmission latency without requiring fine-tuning. Experiments show that our method achieves 74 percent node accuracy on the Replica benchmark, outperforming ConceptGraph. Notably, in the latency-robustness experiment, the LVLM planner assisted by ST-OVSG achieved a planning success rate of 70.5 percent.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。