程序化锚定、具备组合泛化能力的机器人操作
Programmatically Grounded, Compositionally Generalizable Robotic Manipulation
AI总结：
针对传统预训练-微调流程整合视觉-语言模型时动作与视觉信息纠缠、泛化性差的问题，提出模块化框架ProgramPort，通过语义解析生成可执行程序解耦动作与感知，提升机器人操作的零样本和组合泛化能力。
摘要：
在真实世界中运行的机器人既需要丰富的操作技能，也需要具备语义推理能力以判断何时应用这些技能。为实现这一目标，近期研究将大规模预训练视觉-语言（VL）模型的语义表示整合到操作模型中，赋予其更强的通用推理能力。然而，我们发现，用于整合此类表示的传统预训练-微调流程会将领域特定的动作信息学习与领域通用的视觉信息学习纠缠在一起，导致训练数据效率低下，且对未见过的物体和任务泛化性较差。为此，我们提出ProgramPort，这是一种模块化方法，通过利用语言指令的句法和语义结构来更好地利用预训练VL模型。我们的框架使用语义解析器恢复可执行程序，该程序由跨不同模态、基于视觉和动作的功能模块组成。每个功能模块由确定性计算和可学习神经网络组合实现。程序执行会为机器人末端执行器的通用操作原语生成参数。整个模块化网络可通过端到端模仿学习目标进行训练。实验表明，我们的模型成功解耦了动作与感知，在多种操作行为中提升了零样本和组合泛化能力。项目网页：https://progport.github.io。
英文摘要：
Robots operating in the real world require both rich manipulation skills as well as the ability to semantically reason about when to apply those skills. Towards this goal, recent works have integrated semantic representations from large-scale pretrained vision-language (VL) models into manipulation models, imparting them with more general reasoning capabilities. However, we show that the conventional pretraining-finetuning pipeline for integrating such representations entangles the learning of domain-specific action information and domain-general visual information, leading to less data-efficient training and poor generalization to unseen objects and tasks. To this end, we propose ProgramPort, a modular approach to better leverage pretrained VL models by exploiting the syntactic and semantic structures of language instructions. Our framework uses a semantic parser to recover an executable program, composed of functional modules grounded on vision and action across different modalities. Each functional module is realized as a combination of deterministic computation and learnable neural networks. Program execution produces parameters to general manipulation primitives for a robotic end-effector. The entire modular network can be trained with end-to-end imitation learning objectives. Experiments show that our model successfully disentangles action and perception, translating to improved zero-shot and compositional generalization in a variety of manipulation behaviors. Project webpage at: \url{https://progport.github.io}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
