通过多模态演示学习实现人类与机器人动作的对齐
Toward Aligning Human and Robot Actions via Multi-Modal Demonstration Learning
AI总结：
针对人机协作与模仿学习中的动作对应理解问题，提出结合ResNet视觉编码和Perceiver Transformer的多模态演示学习框架，在RH20T数据集上实现人类意图建模与机器人动作预测，提升人机决策对齐能力。
摘要：
理解人类与机器人之间的动作对应关系对于评估决策对齐至关重要，尤其在非结构化环境中的人机协作和模仿学习场景中。我们提出一种多模态演示学习框架，显式地对RGB视频中的人类演示与体素化RGB-D空间中的机器人演示进行建模。针对RH20T数据集的“拾取与放置”任务，我们利用5名用户在10个不同场景中的数据，结合基于ResNet的视觉编码（用于人类意图建模）和Perceiver Transformer（用于体素化机器人动作预测）。经过2000个训练轮次后，人类模型准确率达71.67%，机器人模型准确率达71.8%，证明该框架在复杂多模态人机操作行为对齐中的潜力。
英文摘要：
Understanding action correspondence between humans and robots is essential for evaluating alignment in decision-making, particularly in human-robot collaboration and imitation learning within unstructured environments. We propose a multimodal demonstration learning framework that explicitly models human demonstrations from RGB video with robot demonstrations in voxelized RGB-D space. Focusing on the "pick and place" task from the RH20T dataset, we utilize data from 5 users across 10 diverse scenes. Our approach combines ResNet-based visual encoding for human intention modeling and a Perceiver Transformer for voxel-based robot action prediction. After 2000 training epochs, the human model reaches 71.67% accuracy, and the robot model achieves 71.8% accuracy, demonstrating the framework's potential for aligning complex, multimodal human and robot behaviors in manipulation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
