图融合视觉-语言-动作用于多臂机器人操作中的策略推理
Graph-Fused Vision-Language-Action for Policy Reasoning in Multi-Arm Robotic Manipulation
AI总结：
提出图融合视觉-语言-动作框架GF-VLA，从RGB-D人类演示中提取交互线索构建场景图，结合语言条件变换器生成行为树和运动原语，实现双臂任务级推理与执行，在四个基准上达到高准确率和成功率。
摘要：
从人类视频演示中获取灵巧的机器人技能仍然是一个重大挑战，这主要归因于传统上对低级轨迹复制的依赖，而这种复制往往无法在不同物体、空间布局和操作器配置之间进行泛化。为了解决这一局限性，我们引入了图融合视觉-语言-动作（GF-VLA），这是一个统一的框架，使双臂机器人系统能够直接从RGB-D人类演示中执行任务级推理和执行。GF-VLA采用信息论方法来提取任务相关线索，有选择地突出关键的手-物体和物体-物体交互。这些线索被结构化为按时间排序的场景图，随后与语言条件变换器集成，以产生层次化行为树和可解释的笛卡尔运动原语。为了提高双臂执行效率，我们提出了一种跨臂分配策略，该策略无需显式几何建模即可自主确定夹爪分配。我们在四个涉及符号结构构建和空间泛化的双臂积木组装基准上验证了GF-VLA。实证结果表明，所提出的表示实现了超过95%的图准确率和93%的子任务分割，使语言-动作规划器能够生成稳健、可解释的任务策略。当部署在双臂机器人上时，这些策略在堆叠、字母形成和几何重配置任务中达到了94%的抓取可靠性、89%的放置准确率和90%的整体任务成功率，证明了在多样空间和语义变化下的强泛化性和稳健性。
英文摘要：
Acquiring dexterous robotic skills from human video demonstrations remains a significant challenge, largely due to conventional reliance on low-level trajectory replication, which often fails to generalize across varying objects, spatial layouts, and manipulator configurations. To address this limitation, we introduce Graph-Fused Vision-Language-Action (GF-VLA), a unified framework that enables dual-arm robotic systems to perform task-level reasoning and execution directly from RGB-D human demonstrations. GF-VLA employs an information-theoretic approach to extract task-relevant cues, selectively highlighting critical hand-object and object-object interactions. These cues are structured into temporally ordered scene graphs, which are subsequently integrated with a language-conditioned transformer to produce hierarchical behavior trees and interpretable Cartesian motion primitives. To enhance efficiency in bimanual execution, we propose a cross-arm allocation strategy that autonomously determines gripper assignment without requiring explicit geometric modeling. We validate GF-VLA on four dual-arm block assembly benchmarks involving symbolic structure construction and spatial generalization. Empirical results demonstrate that the proposed representation achieves over 95% graph accuracy and 93% subtask segmentation, enabling the language-action planner to generate robust, interpretable task policies. When deployed on a dual-arm robot, these policies attain 94% grasp reliability, 89% placement accuracy, and 90% overall task success across stacking, letter-formation, and geometric reconfiguration tasks, evidencing strong generalization and robustness under diverse spatial and semantic variations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
