使用深度强化学习实现软驱动昆虫级微型飞行器的悬停飞行
Hovering Flight of Soft-Actuated Insect-Scale Micro Aerial Vehicles using Deep Reinforcement Learning
AI总结：
本文提出结合状态-动作重匹配、域随机化行为克隆与PPO微调的深度RL控制器，实现软驱动昆虫级微型飞行器的零样本稳定悬停。
摘要：
软驱动昆虫级微型飞行器（IMAV）为设计鲁棒且计算高效的控制器带来了独特挑战。在毫米尺度下，快速的机器人动力学（约毫秒级）连同系统延迟、模型不确定性和外部扰动，会显著影响飞行性能。本文设计了一种深度强化学习（RL）控制器，用于应对系统延迟和不确定性。为初始化该神经网络（NN）控制器，我们提出一种改进的行为克隆（BC）方法：通过状态-动作重新匹配来处理延迟，并利用域随机化专家示教来应对不确定性。随后，在强化学习过程中采用近端策略优化（PPO）对策略进行微调，以提升性能并平滑控制指令。仿真中，与基线BC相比，我们的改进BC显著提高了平均奖励；基于PPO的强化学习改善了飞行质量并减少了指令波动。我们将该控制器部署到两台重量分别为720 mg和850 mg的不同昆虫级飞行机器人上。机器人完成了多次成功的零样本悬停飞行，最长持续50秒，横向均方根误差为1.34 cm，高度均方根误差为0.05 cm，标志着软驱动IMAV上首次实现基于深度RL的端到端飞行。
英文摘要：
Soft-actuated insect-scale micro aerial vehicles (IMAVs) pose unique challenges for designing robust and computationally efficient controllers. At the millimeter scale, fast robot dynamics ($\sim$ms), together with system delay, model uncertainty, and external disturbances significantly affect flight performances. Here, we design a deep reinforcement learning (RL) controller that addresses system delay and uncertainties. To initialize this neural network (NN) controller, we propose a modified behavior cloning (BC) approach with state-action re-matching to account for delay and domain-randomized expert demonstration to tackle uncertainty. Then we apply proximal policy optimization (PPO) to fine-tune the policy during RL, enhancing performance and smoothing commands. In simulations, our modified BC substantially increases the mean reward compared to baseline BC; and RL with PPO improves flight quality and reduces command fluctuations. We deploy this controller on two different insect-scale aerial robots that weigh 720 mg and 850 mg, respectively. The robots demonstrate multiple successful zero-shot hovering flights, with the longest lasting 50 seconds and root-mean-square errors of 1.34 cm in lateral direction and 0.05 cm in altitude, marking the first end-to-end deep RL-based flight on soft-driven IMAVs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
