基于强化学习的导弹鲁棒自动驾驶仪设计
Reinforcement Learning for Robust Missile Autopilot Design
摘要：
由于导弹的飞行包线大，飞行动力学非线性，导弹自动驾驶仪控制器的设计一直是一项复杂的任务。虽然控制理论经常涉及参数的调度过程，但强化学习已经在越来越复杂的任务中呈现出有趣的结果，从视频游戏到具有连续动作域的机器人任务。然而，它仍然缺乏关于如何找到足够的奖励函数和探索策略的更清晰的见解。据我们所知，这项工作是提出强化学习作为飞行控制框架的先驱。事实上，它的目的是训练一个无模型的代理，可以控制导弹的纵向飞行，实现最佳性能和鲁棒性的不确定性。为此，根据TRPO的方法，收集的经验根据HER进行扩充，存储在重放缓冲区中，并根据其重要性进行采样。这项工作不仅增强了优先经验重放到BPER的概念，而且还重新制定HER，只有当训练进度收敛到次优策略时，才激活它们，这被称为SER方法。此外，奖励工程过程是仔细详细的。结果表明，它是可能的，以实现最佳的性能和提高代理的鲁棒性的不确定性（与标称性能的低损害），通过进一步训练它在非标称环境中，因此验证所提出的方法，并鼓励未来的研究在这一领域。
英文摘要：
Designing missiles' autopilot controllers has been a complex task, given the extensive flight envelope and the nonlinear flight dynamics. A solution that can excel both in nominal performance and in robustness to uncertainties is still to be found. While Control Theory often debouches into parameters' scheduling procedures, Reinforcement Learning has presented interesting results in ever more complex tasks, going from videogames to robotic tasks with continuous action domains. However, it still lacks clearer insights on how to find adequate reward functions and exploration strategies. To the best of our knowledge, this work is pioneer in proposing Reinforcement Learning as a framework for flight control. In fact, it aims at training a model-free agent that can control the longitudinal flight of a missile, achieving optimal performance and robustness to uncertainties. To that end, under TRPO's methodology, the collected experience is augmented according to HER, stored in a replay buffer and sampled according to its significance. Not only does this work enhance the concept of prioritized experience replay into BPER, but it also reformulates HER, activating them both only when the training progress converges to suboptimal policies, in what is proposed as the SER methodology. Besides, the Reward Engineering process is carefully detailed. The results show that it is possible both to achieve the optimal performance and to improve the agent's robustness to uncertainties (with low damage on nominal performance) by further training it in non-nominal environments, therefore validating the proposed approach and encouraging future research in this field.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
