Distillation-PPO：一种用于人形机器人感知运动的新型两阶段强化学习框架
Distillation-PPO: A Novel Two-Stage Reinforcement Learning Framework for Humanoid Robot Perceptive Locomotion
AI总结：
本文提出Distillation-PPO两阶段框架，结合MDP教师策略监督与POMDP强化学习，提升人形机器人感知运动的训练效率、稳定性及实际泛化能力。
摘要：
近年来，人形机器人因其对环境的高适应性和类人特征而受到学术界和工业界的广泛关注。随着强化学习的快速发展，人形机器人的行走控制取得了显著进展。然而，现有方法在处理复杂环境和不规则地形时仍面临挑战。在感知运动领域，现有方法通常分为两阶段方法和端到端方法。两阶段方法首先在仿真环境中训练教师策略，然后使用蒸馏技术（如DAgger）将作为潜在特征或动作学习到的特权信息迁移给学生策略。端到端方法则放弃对特权信息的学习，直接通过强化学习从部分可观测马尔可夫决策过程（POMDP）中学习策略。然而，由于缺乏教师策略的监督，端到端方法往往面临训练困难，并在实际应用中表现出不稳定的性能。本文提出了一种创新的两阶段感知运动框架，该框架结合了在完全可观测马尔可夫决策过程（MDP）中学习到的教师策略的优势，以正则化和监督学生策略。同时，它利用强化学习的特性，确保学生策略能够在POMDP中持续学习，从而提升模型的上限。我们的实验结果表明，所提出的两阶段训练框架在仿真环境中实现了更高的训练效率和稳定性，同时在实际应用中表现出更好的鲁棒性和泛化能力。
英文摘要：
In recent years, humanoid robots have garnered significant attention from both academia and industry due to their high adaptability to environments and human-like characteristics. With the rapid advancement of reinforcement learning, substantial progress has been made in the walking control of humanoid robots. However, existing methods still face challenges when dealing with complex environments and irregular terrains. In the field of perceptive locomotion, existing approaches are generally divided into two-stage methods and end-to-end methods. Two-stage methods first train a teacher policy in a simulated environment and then use distillation techniques, such as DAgger, to transfer the privileged information learned as latent features or actions to the student policy. End-to-end methods, on the other hand, forgo the learning of privileged information and directly learn policies from a partially observable Markov decision process (POMDP) through reinforcement learning. However, due to the lack of supervision from a teacher policy, end-to-end methods often face difficulties in training and exhibit unstable performance in real-world applications. This paper proposes an innovative two-stage perceptive locomotion framework that combines the advantages of teacher policies learned in a fully observable Markov decision process (MDP) to regularize and supervise the student policy. At the same time, it leverages the characteristics of reinforcement learning to ensure that the student policy can continue to learn in a POMDP, thereby enhancing the model's upper bound. Our experimental results demonstrate that our two-stage training framework achieves higher training efficiency and stability in simulated environments, while also exhibiting better robustness and generalization capabilities in real-world applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
