基于少样本元离线强化学习的弹性无人机轨迹规划
Resilient UAV Trajectory Planning via Few-Shot Meta-Offline Reinforcement Learning
AI总结：
本文提出一种结合CQL与MAML的少样本元离线强化学习算法，用于无人机轨迹与调度优化，在无需在线交互的情况下实现更快收敛、最优AoI与功率联合优化，并对环境变化具有弹性。
摘要：
强化学习（RL）在未来的超5G和6G系统中一直是一种有前景的核心方法。其主要优势在于能够在复杂且高维的无线环境中进行鲁棒的免模型决策。然而，现有的大多数RL框架依赖与环境的在线交互，由于安全和成本方面的考虑，这种交互可能不可行。在线RL的另一个问题是所设计算法在动态或新环境中缺乏可扩展性。本文提出了一种新颖的、具有弹性的少样本元离线RL算法，该算法结合了使用保守Q学习（CQL）的离线RL和使用模型无关元学习（MAML）的元学习。所提算法能够利用静态离线数据集训练RL模型，而无需与环境进行任何在线交互。此外，借助MAML，所提模型可以扩展到新的未见环境中。我们展示了所提算法用于优化无人机（UAV）的轨迹和调度策略，以最小化信息年龄（AoI）和有限功率设备的发射功率。数值结果表明，所提出的少样本元离线RL算法比深度Q网络和CQL等基线方案收敛更快。此外，它是唯一能够利用仅含少量数据点的离线数据集实现AoI和发射功率联合最优，并且对因前所未有的环境变化而导致的网络故障具有弹性的算法。
英文摘要：
Reinforcement learning (RL) has been a promising essence in future 5G-beyond and 6G systems. Its main advantage lies in its robust model-free decision-making in complex and large-dimension wireless environments. However, most existing RL frameworks rely on online interaction with the environment, which might not be feasible due to safety and cost concerns. Another problem with online RL is the lack of scalability of the designed algorithm with dynamic or new environments. This work proposes a novel, resilient, few-shot meta-offline RL algorithm combining offline RL using conservative Q-learning (CQL) and meta-learning using model-agnostic meta-learning (MAML). The proposed algorithm can train RL models using static offline datasets without any online interaction with the environments. In addition, with the aid of MAML, the proposed model can be scaled up to new unseen environments. We showcase the proposed algorithm for optimizing an unmanned aerial vehicle (UAV) 's trajectory and scheduling policy to minimize the age-of-information (AoI) and transmission power of limited-power devices. Numerical results show that the proposed few-shot meta-offline RL algorithm converges faster than baseline schemes, such as deep Q-networks and CQL. In addition, it is the only algorithm that can achieve optimal joint AoI and transmission power using an offline dataset with few shots of data points and is resilient to network failures due to unprecedented environmental changes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
