人机交互：从视频演示中学习机器人模仿
Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
AI总结：
本研究提出了一种基于视频演示的机器人模仿学习方法，通过模块化框架结合时间位移模块和深度强化学习，实现机器人从无结构视频中学习基本操作技能。
摘要：
学习从演示（LfD）提供了一种有前途的机器人技能获取范式。最近的方法试图直接从视频演示中提取操作指令，但面临两个关键挑战：（1）通用视频字幕模型优先考虑全局场景特征而非任务相关的物体，产生不适合精确机器人执行的描述；（2）将视觉理解与策略学习耦合的端到端架构需要大量的配对数据集，并且在物体和场景之间难以泛化。为了解决这些限制，我们提出了一种新的“人机”模仿学习流程，使机器人能够直接从无结构的视频演示中获取操作技能，受到人类通过观看和模仿学习能力的启发。我们的关键创新是一个模块化框架，将学习过程分为两个不同的阶段：（1）视频理解，结合时间位移模块（TSM）与视觉语言模型（VLMs）来提取动作并识别交互物体；（2）机器人模仿，采用基于TD3的深度强化学习执行演示的操作。我们在PyBullet模拟环境中的UR5e机械臂和现实世界实验中的UF850机械臂上验证了我们的方法，涵盖了四种基本动作：到达、抓取、移动和放置。在视频理解方面，我们的方法在标准物体上实现了89.97%的动作分类准确率和BLEU-4得分为0.351，在新物体上得分为0.265，分别比最佳基线提高了76.4%和128.4%。在机器人操作方面，我们的框架在所有动作上的平均成功率为87.5%，在到达任务上达到100%的成功率，在复杂的抓取和放置操作上达到最高90%。项目网站可在https://thanhnguyencanh.github.io/LfD4hri上获取。
英文摘要：
Learning from Demonstration (LfD) offers a promising paradigm for robot skill acquisition. Recent approaches attempt to extract manipulation commands directly from video demonstrations, yet face two critical challenges: (1) general video captioning models prioritize global scene features over task-relevant objects, producing descriptions unsuitable for precise robotic execution, and (2) end-to-end architectures coupling visual understanding with policy learning require extensive paired datasets and struggle to generalize across objects and scenarios. To address these limitations, we propose a novel ``Human-to-Robot'' imitation learning pipeline that enables robots to acquire manipulation skills directly from unstructured video demonstrations, inspired by the human ability to learn by watching and imitating. Our key innovation is a modular framework that decouples the learning process into two distinct stages: (1) Video Understanding, which combines Temporal Shift Modules (TSM) with Vision-Language Models (VLMs) to extract actions and identify interacted objects, and (2) Robot Imitation, which employs TD3-based deep reinforcement learning to execute the demonstrated manipulations. We validated our approach in PyBullet simulation environments with a UR5e manipulator and in a real-world experiment with a UF850 manipulator across four fundamental actions: reach, pick, move, and put. For video understanding, our method achieves 89.97% action classification accuracy and BLEU-4 scores of 0.351 on standard objects and 0.265 on novel objects, representing improvements of 76.4% and 128.4% over the best baseline, respectively. For robot manipulation, our framework achieves an average success rate of 87.5% across all actions, with 100% success on reaching tasks and up to 90% on complex pick-and-place operations. The project website is available at https://thanhnguyencanh.github.io/LfD4hri.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
