面向VTOL-UAV海上对接作业的仿真到现实策略迁移深度强化学习
Deep Reinforcement Learning for Sim-to-Real Policy Transfer of VTOL-UAVs Offshore Docking Operations
AI总结：
本文提出一种面向VTOL-UAV海上对接的深度强化学习方法，将着陆分解为接近与着陆两阶段，结合JONSWAP波浪模型训练PPO等智能体，提升仿真到现实迁移成功率。
摘要：
本文提出了一种新颖的强化学习（RL）方法，用于垂直起降无人机（VTOL-UAV）的仿真到现实策略迁移。所提方法面向海上作业中VTOL-UAV在海上对接站着陆的场景。海上作业中的VTOL-UAV在作战半径方面受到限制，主要源于其电池容量带来的约束。在充电平台上自主着陆的概念为缓解这些限制提供了有吸引力的前景，因为它能够促进电池充电和数据传输。然而，当前的深度强化学习（DRL）方法存在训练时间长和成功率不高等缺点。本文通过将着陆过程分解为一系列更易处理但相互类似的任务，即接近阶段和着陆阶段，全面应对这些问题。所提架构在接近阶段采用基于模型的控制方案，此时VTOL-UAV正在接近海上对接站。在着陆阶段，DRL智能体被离线训练，以学习在海上站点对接的最优策略。联合北海波浪计划（JONSWAP）谱模型被用于为每个回合生成波浪模型，从而增强面向sim2real迁移的策略泛化能力。通过数值仿真测试了一组DRL算法，包括基于价值的智能体和基于策略的智能体，例如深度Q网络（DQN）和近端策略优化（PPO）。数值实验表明，PPO智能体能够学习复杂且高效的策略，以在不确定环境中着陆，从而提高了成功实现仿真到现实迁移的可能性。
英文摘要：
This paper proposes a novel Reinforcement Learning (RL) approach for sim-to-real policy transfer of Vertical Take-Off and Landing Unmanned Aerial Vehicle (VTOL-UAV). The proposed approach is designed for VTOL-UAV landing on offshore docking stations in maritime operations. VTOL-UAVs in maritime operations encounter limitations in their operational range, primarily stemming from constraints imposed by their battery capacity. The concept of autonomous landing on a charging platform presents an intriguing prospect for mitigating these limitations by facilitating battery charging and data transfer. However, current Deep Reinforcement Learning (DRL) methods exhibit drawbacks, including lengthy training times, and modest success rates. In this paper, we tackle these concerns comprehensively by decomposing the landing procedure into a sequence of more manageable but analogous tasks in terms of an approach phase and a landing phase. The proposed architecture utilizes a model-based control scheme for the approach phase, where the VTOL-UAV is approaching the offshore docking station. In the Landing phase, DRL agents were trained offline to learn the optimal policy to dock on the offshore station. The Joint North Sea Wave Project (JONSWAP) spectrum model has been employed to create a wave model for each episode, enhancing policy generalization for sim2real transfer. A set of DRL algorithms have been tested through numerical simulations including value-based agents and policy-based agents such as Deep \textit{Q} Networks (DQN) and Proximal Policy Optimization (PPO) respectively. The numerical experiments show that the PPO agent can learn complicated and efficient policies to land in uncertain environments, which in turn enhances the likelihood of successful sim-to-real transfer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
