MPC-Injection: 将离线运动学RL偏向控制器诱导的行为盆地
MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins
AI总结：
提出MPC-Injection方法，通过将模型预测控制器的轨迹注入回放缓冲区，引导强化学习策略学习期望步态，无需奖励塑形或对抗模仿学习，在仿真和真实机器人上验证有效性。
摘要：
运动强化学习（RL）经常收敛到局部最优但不可部署的行为，例如肢体振动或躯干滑动，这些行为最大化回报但无法产生可用的步态。我们提出MPC-Injection，一种低开销方法，通过将从解决相同马尔可夫决策过程的模型预测控制器获得的转移插入回放缓冲区，将RL引导向设计者偏好的步态。与奖励塑形不同，MPC-Injection不需要重新设计任务奖励；与对抗模仿学习不同，它不添加判别器、运动学重定向或辅助目标。相反，控制器的偏好行为纯粹通过回放状态分布传递给策略。在仿真中的2D步行器上以及通过在Go2四足机器人上的仿真到现实评估，我们展示了MPC-Injection使用一到两项任务奖励将策略驱动到控制器的行为盆地，产生与使用二十一项调优奖励的奖励塑形以及不使用判别器和重定向开销的对抗运动先验在质量上可比的步态。我们进一步分析了注入的转移如何将演员-评论家更新偏向控制器访问的状态，使得策略能够学习在简单奖励函数下纯RL可能无法达到的行为。
英文摘要：
Reinforcement learning (RL) for locomotion frequently converges to locally optimal but undeployable behaviors, such as vibrating limbs or scooting on the torso, that maximize return without producing a usable gait. We present MPC-Injection, a low-overhead method that steers RL toward a designer-preferred behavior by inserting transitions generated in the same environment by a model predictive controller (MPC). Unlike reward shaping, MPC-Injection does not require redesigning the task reward, and unlike adversarial imitation learning, it adds no discriminator, no kinematic retargeting, and no auxiliary objective. We analyze how the injected transitions bias the learning, allowing the policy to converge to behaviors that pure RL may fail to reach under simple reward functions. On a 2D walker in simulation and with sim-to-real evaluation on a Go2 quadruped, we show that MPC-Injection produces gaits qualitatively comparable to those of reward shaping and adversarial motion priors. We also show that MPC-Injection can complete a barrel roll that pure RL fails to achieve under the same simple reward and can select between trotting and bounding gaits only through changing the injected MPC data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
