从多模态输入生成物理真实且可定向的人体运动
Generating Physically Realistic and Directable Human Motions from Multi-Modal Inputs
AI总结:
针对多模态稀疏输入生成物理真实人体运动的需求,提出掩码类人机器人控制器(MHC),通过多目标模仿学习实现输入追赶、技能切换与故障恢复,在87种技能数据集上验证并展示多模态应用能力。
摘要:
本工作聚焦于从多模态输入生成基于物理的真实人体行为,这些输入可能仅部分指定所需运动。例如,输入可能来自提供手臂运动和身体速度的VR控制器、部分关键点动画、应用于视频的计算机视觉,甚至更高级的运动目标。这需要一个多功能的低级别类人机器人控制器,能够处理此类稀疏、未充分指定的引导,无缝切换技能,并从故障中恢复。当前从演示数据学习类人机器人控制器的方法具备部分这些特征,但没有一种能全部实现。为此,我们引入掩码类人机器人控制器(MHC),这是一种在增强且选择性掩码的运动演示上应用多目标模仿学习的新方法。该训练方法使MHC具备关键能力:追赶不同步的输入命令、结合多个运动序列的元素,以及从稀疏多模态输入完成未指定的运动部分。我们展示了在包含87种不同技能的数据集上学习的MHC的这些关键能力,并展示了不同的多模态用例,包括与规划框架集成,以突出MHC无需微调即可解决新用户定义任务的能力。
英文摘要:
This work focuses on generating realistic, physically-based human behaviors from multi-modal inputs, which may only partially specify the desired motion. For example, the input may come from a VR controller providing arm motion and body velocity, partial key-point animation, computer vision applied to videos, or even higher-level motion goals. This requires a versatile low-level humanoid controller that can handle such sparse, under-specified guidance, seamlessly switch between skills, and recover from failures. Current approaches for learning humanoid controllers from demonstration data capture some of these characteristics, but none achieve them all. To this end, we introduce the Masked Humanoid Controller (MHC), a novel approach that applies multi-objective imitation learning on augmented and selectively masked motion demonstrations. The training methodology results in an MHC that exhibits the key capabilities of catch-up to out-of-sync input commands, combining elements from multiple motion sequences, and completing unspecified parts of motions from sparse multimodal input. We demonstrate these key capabilities for an MHC learned over a dataset of 87 diverse skills and showcase different multi-modal use cases, including integration with planning frameworks to highlight MHC's ability to solve new user-defined tasks without any finetuning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。