用于追逃博弈中敏捷四旋翼的学习控制器
Learned Controllers for Agile Quadrotors in Pursuit-Evasion Games
AI总结：
针对四旋翼追逃博弈中的非平稳性、灾难性遗忘和策略循环问题，提出异步多阶段种群训练与Hedge采样方法，通过策略空间响应或acles与PPO训练，实现新策略发现与旧策略保持，并验证了硬件上的零样本迁移。
摘要：
在这封信中，我们研究了1v1四旋翼追逃博弈，其中追捕者和逃避者通过相互对抗的强化学习（RL）进行训练。这种对抗性设置面临众所周知的挑战：每个智能体的策略在训练过程中发生变化，造成非平稳环境；智能体可能过度拟合当前对手而忘记早期策略（灾难性遗忘）；竞争动态可能导致策略循环或策略崩溃。为了解决这些问题，我们提出了异步多阶段种群训练与Hedge采样（AMSPBH），这是一种基于策略空间响应或acles（PSRO）并适用于四旋翼RL控制的方法。PSRO维护一个先前训练策略的种群，并针对该种群的混合而非单个对手训练新的近似最佳响应。在AMSPBH中，每一代使用近端策略优化（PPO）针对冻结的对手策略训练一个智能体，同时Hedge采样器为当前难以击败的对手分配更高的概率。我们表明：（i）AMSPBH在保持对较老对手的能力的同时发现新策略，达到额外最佳响应训练带来有限改进的状态；（ii）与仅针对最新对手训练相比，基于种群的训练在多样化和未见过的策略上泛化更好；（iii）学习到的种群包含不同的追捕和逃避行为，为发现弱点和提高控制器鲁棒性提供了有用的策略多样性。我们通过Crazyflie无刷四旋翼的硬件实验验证了训练策略，展示了针对手工制作和学习的对手的敏捷、反应式追逃行为的零样本仿真到现实迁移，物理飞行速度高达4.87米/秒。
英文摘要：
In this letter we study 1v1 quadrotor pursuit-evasion, where a pursuer and an evader are trained via reinforcement learning (RL) by competing against each other. Such adversarial settings face well-known challenges: each agent's policy changes during training, creating a non-stationary environment; agents might overfit to the current opponent and forget earlier strategies (catastrophic forgetting); and the competitive dynamics can cause strategy cycling or policy collapse. To address these issues, we propose Asynchronous Multi-Stage Population-Based training with Hedge sampling (AMSPBH), a method based on Policy-Space Response Oracles (PSRO) and adapted to quadrotor RL control. PSRO maintains a population of previously trained policies and trains new approximate best responses against mixtures of that population instead of against a single opponent. In AMSPBH, each generation trains one agent with Proximal Policy Optimization (PPO) against frozen opponent policies, while a Hedge sampler assigns higher probability to opponents that are currently difficult to beat. We show that: (i) AMSPBH discovers new strategies while retaining competence against older opponents, reaching a regime where additional best-response training gives limited improvement; (ii) compared to training against only the latest opponent, population-based training generalizes better across diverse and unseen strategies; and (iii) the learned population contains distinct pursuit and evasion behaviors, providing useful strategic diversity for finding weaknesses and improving controller robustness. We validate the trained policies with hardware experiments on Crazyflie brushless quadrotors, showing zero-shot sim-to-real transfer of agile, reactive pursuit-evasion behavior against both handcrafted and learned adversaries, with physical flights reaching up to 4.87 m/s.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
