arXiv 2601.114602026-05-30cs.ROcs.LG

面向双臂操作的语义-几何任务表示：从人类示范到机器人动作规划

Semantic-Geometric Task Representations for Bimanual Manipulation from Human Demonstrations to Robot Action Planning

Interactive Robot Perception & Learning (PEARL) Lab, Computer Science Dept., TU Darmstadt, Germany（图腾大学达姆施塔特分校计算机科学系交互机器人感知与学习实验室）；Hessian.AI, Darmstadt, Germany（黑森人工智能公司）；Robotics Institute Germany (RIG)（德国机器人研究所）；Interactive AI Algorithms & Cognitive Models for Human-AI Interaction (IKIDA), Computer Science Dept., TU Darmstadt, Germany（人机交互的交互人工智能算法与认知模型（IKIDA），图腾大学达姆施塔特分校计算机科学系）；Center for Cognitive Science, TU Darmstadt, Germany（图腾大学达姆施塔特分校认知科学中心）

Franziska Herbert, Vignesh Prasad, Han Liu, Dorothea Koert, Georgia Chalvatzaki

备注：: 9 pages, 7 figures, preprint

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出一种语义-几何图任务表示，通过消息传递神经网络编码器和Transformer解码器联合编码对象身份、语义关系和运动历史，实现从人类示范中学习结构化任务表示，并支持跨实体迁移和双臂操作规划。

摘要：

从人类示范中学习结构化任务表示对于双臂操作至关重要，因为动作顺序、对象参与和交互几何在不同执行中变化显著。一个关键挑战在于以支持任务进展推理的形式，联合捕获离散的语义任务结构和对象中心几何关系的时间演化。我们引入一种基于语义-几何图的任务表示，通过消息传递神经网络（MPNN）编码器和基于Transformer的解码器，联合编码对象身份、对象间语义关系和每个对象的运动历史。编码器仅对时间场景图进行操作，产生与动作标签解耦的结构化表示。解码器则根据动作上下文预测未来动作、关联对象和对象运动。这种解耦学习了任务无关的表示，使得编码器可以通过仅在小型机器人数据集上微调解码器而跨实体复用。在两个数据集的十一个双臂任务中，我们发现结构化语义-几何表示相对于更简单的基于序列模型的优势随着动作顺序和对象参与的任务变异性增加而增长。在部署时，规划器将动作和运动预测与学习的概率运动基元相结合，在两个真实机器人双臂任务上实现了完全任务成功，并优于图消融、Transformer、仅解码器和微调的视觉-语言模型基线。

英文摘要：

Learning structured task representations from human demonstrations is essential for bimanual manipulation, where action ordering, object involvement, and interaction geometry vary significantly across executions. A key challenge lies in jointly capturing the discrete semantic task structure and the temporal evolution of object-centric geometric relations in a form that supports reasoning over task progression. We introduce a semantic--geometric graph-based task representation that jointly encodes object identities, inter-object semantic relations, and per-object motion histories, via a Message Passing Neural Network (MPNN) encoder and a Transformer-based decoder. The encoder operates solely on the temporal scene graph, producing structured representations decoupled from action labels. The decoder then conditions on action-context to forecast future actions, associated objects, and object motions. This decoupling learns task-agnostic representations, enabling encoder reuse across embodiments through decoder-only finetuning on a small robot dataset. Across eleven bimanual tasks from two datasets, we find that the benefit of structured semantic--geometric representations over simpler sequence-based models grows with task variability in action ordering and object involvement. At deployment, a planner couples the action and motion predictions with learned Probabilistic Movement Primitives, achieving full task success on two real-robot bimanual tasks and outperforming graph ablations, Transformer, decoder-only, and finetuned vision-language model baselines.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。