机器人助手:利用灵巧视觉-语言-动作模型完成协作任务
Robotic Assistant: Completing Collaborative Tasks with Dexterous Vision-Language-Action Models
AI总结:
本研究通过为预训练VLA模型Open-VLA添加FiLM条件调制、辅助意图头和动作空间后处理,实现了低提示成本的灵巧人机协作,验证了增量动作的有效性并明确了各模块的性能贡献。
摘要:
我们对预训练的视觉-语言-动作(Vision-Language-Action,VLA)模型Open-VLA进行适配,以最少的语言提示实现灵巧的人机协作。我们的方法新增了三项设计:(i)在视觉骨干网络中加入FiLM条件调制,实现任务感知的感知;(ii)辅助意图头,用于预测协作者的手部姿态和目标线索;(iii)动作空间后处理,先预测紧凑的增量(位置/旋转)和经PCA降维的手指关节,再映射为完整指令。\n我们使用多视角遥操作的Franka机器人与Mimic-hand数据集(经MediaPipe手部姿态增强)开展实验,证明增量动作表现稳定,4个主成分可解释约96%的手部关节方差。消融实验显示,动作后处理是性能提升的主要驱动因素;辅助意图头有帮助,FiLM的效果好坏参半,方向运动损失则会损害性能。我们构建的实时栈(单张RTX 4090上延迟约0.3秒)可将“拾取”和“传递”组合成长时序行为。研究还发现,模型对特定演示者的“训练者过拟合”是核心局限。
英文摘要:
We adapt a pre-trained Vision-Language-Action (VLA) model (Open-VLA) for dexterous human-robot collaboration with minimal language prompting. Our approach adds (i) FiLM conditioning to visual backbones for task-aware perception, (ii) an auxiliary intent head that predicts collaborator hand pose and target cues, and (iii) action-space post-processing that predicts compact deltas (position/rotation) and PCA-reduced finger joints before mapping to full commands. Using a multi-view, teleoperated Franka and Mimic-hand dataset augmented with MediaPipe hand poses, we demonstrate that delta actions are well-behaved and that four principal components explain ~96% of hand-joint variance. Ablations identify action post-processing as the primary performance driver; auxiliary intent helps, FiLM is mixed, and a directional motion loss is detrimental. A real-time stack (~0.3 s latency on one RTX 4090) composes "pick-up" and "pass" into a long-horizon behavior. We surface "trainer overfitting" to specific demonstrators as the key limitation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。