arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.01366cs.ROcs.CV
收藏

顺应阶段得分:用于机器人操作的阶段引导动态多感官融合

Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation

Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Shenzhen Taobotics Co., Ltd.(深圳桃机器人有限公司);Institute of Artificial Intelligence (TeleAI), China Telecom(中国电信人工智能研究院(TeleAI))

Ruoxuan Feng, Di Hu, Wenke Ma, Xuelong Li

AI总结:

针对机器人操作中多感官信息需随任务阶段动态调整的问题,提出 MS-Bot,利用由粗到细的阶段理解引导视觉、听觉和触觉融合,并在倾倒与键槽销钉插入任务中验证了更有效、可解释且贴近人类的融合表现。

摘要:

人类在与环境互动时,具有灵活切换不同感官的卓越能力。想象一位厨师熟练地根据颜色、声音和香气判断加入食材的时机并控制火候,无缝穿行于复杂烹饪过程的每个阶段。这种能力建立在对任务阶段的透彻理解之上,因为实现每个阶段内的子目标可能需要运用不同的感官。为了赋予机器人类似能力,我们将按子目标划分的任务阶段纳入模仿学习过程,以相应地引导动态多感官融合。我们提出了 MS-Bot,这是一种具有由粗到细阶段理解能力的阶段引导动态多感官融合方法,它根据所预测当前阶段内的细粒度状态动态调整各模态的优先级。我们训练了一个配备视觉、听觉和触觉传感器的机器人系统,以完成具有挑战性的机器人操作任务:倾倒和带键槽的销钉插入。实验结果表明,我们的方法能够实现更有效、更可解释的动态融合,并且与现有方法相比更贴近人类的融合过程。

英文摘要:

Humans possess a remarkable talent for flexibly alternating to different senses when interacting with the environment. Picture a chef skillfully gauging the timing of ingredient additions and controlling the heat according to the colors, sounds, and aromas, seamlessly navigating through every stage of the complex cooking process. This ability is founded upon a thorough comprehension of task stages, as achieving the sub-goal within each stage can necessitate the utilization of different senses. In order to endow robots with similar ability, we incorporate the task stages divided by sub-goals into the imitation learning process to accordingly guide dynamic multi-sensory fusion. We propose MS-Bot, a stage-guided dynamic multi-sensory fusion method with coarse-to-fine stage understanding, which dynamically adjusts the priority of modalities based on the fine-grained state within the predicted current stage. We train a robot system equipped with visual, auditory, and tactile sensors to accomplish challenging robotic manipulation tasks: pouring and peg insertion with keyway. Experimental results indicate that our approach enables more effective and explainable dynamic fusion, aligning more closely with the human fusion process than existing methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑