通过多任务参考和目标驱动的强化学习框架从参考中泛化
Generalizing from References using a Multi-Task Reference and Goal-Driven RL Framework
AI总结:
本文提出一种多任务强化学习框架,通过参考和目标驱动的方法,从密集参考监督中学习结构化的运动技能,并实现对新目标和初始条件的适应。
摘要:
从人类动作学习敏捷的人形行为提供了一种强大的途径来实现自然协调的控制,但现有方法面临一个持久的权衡:参考跟踪策略往往在演示数据集之外容易崩溃,而纯粹的任务驱动强化学习(RL)虽然可以实现适应性,但会以运动质量为代价。我们引入了一个统一的多任务RL框架,通过将参考运动作为行为塑造的先验而非部署时的约束来弥合这一差距。一个单一的目标条件政策在两个共享相同观测和动作空间但初始化方案、命令空间和奖励结构不同的任务上联合训练:(i)一个参考引导的模仿任务,其中参考轨迹定义密集的模仿奖励但不作为策略输入提供,以及(ii)一个目标条件的泛化任务,其中目标独立于任何参考采样,奖励仅反映任务成功。通过在共享的框架内共同优化这些目标,策略从密集的参考监督中获得结构化的、人类般的运动技能,同时学习将这些技能适应到新的目标和初始条件。这在不使用对抗性目标、显式轨迹跟踪、相位变量或参考依赖推断的情况下实现。我们评估了该方法在具有多样化运动行为(例如跳跃和攀爬)的挑战性基于盒子的公园游乐场上的表现,并展示了学习的控制器在超越参考分布的同时保持运动的自然性。最后,我们通过组合多个学习到的技能来演示长周期行为生成,展示了在复杂场景中学习到的策略的灵活性。
英文摘要:
Learning agile humanoid behaviors from human motion offers a powerful route to natural, coordinated control, but existing approaches face a persistent trade-off: reference-tracking policies are often brittle outside the demonstration dataset, while purely task-driven Reinforcement Learning (RL) can achieve adaptability at the cost of motion quality. We introduce a unified multi-task RL framework that bridges this gap by treating reference motion as a prior for behavioral shaping rather than a deployment-time constraint. A single goal-conditioned policy is trained jointly on two tasks that share the same observation and action spaces, but differ in their initialization schemes, command spaces, and reward structures: (i) a reference-guided imitation task in which reference trajectories define dense imitation rewards but are not provided as policy inputs, and (ii) a goal-conditioned generalization task in which goals are sampled independently of any reference and where rewards reflect only task success. By co-optimizing these objectives within a shared formulation, the policy acquires structured, human-like motor skills from dense reference supervision while learning to adapt these skills to novel goals and initial conditions. This is achieved without adversarial objectives, explicit trajectory tracking, phase variables, or reference-dependent inference. We evaluate the method on a challenging box-based parkour playground that demands diverse athletic behaviors (e.g., jumping and climbing), and show that the learned controller transfers beyond the reference distribution while preserving motion naturalness. Finally, we demonstrate long-horizon behavior generation by composing multiple learned skills, illustrating the flexibility of the learned polices in complex scenarios.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。