RAPID-RL：一种用于高效深度强化学习的可重配置结构
RAPID-RL: A Reconfigurable Architecture with Preemptive-Exits for Efficient Deep-Reinforcement Learning
摘要：
当今的深度强化学习（RL）系统在构建超越人类水平的智能代理方面表现出了巨大的潜力。然而，与底层深度神经网络（DNN）相关联的计算复杂性导致功耗高的实现。这使得深度RL系统不适合部署在资源受限的边缘设备上。为了应对这一挑战，我们提出了一种可重配置的架构，具有抢占式出口，以实现高效的深度RL（RAPID-RL）。RAPID-RL支持根据输入的难度级别有条件激活DNN层。这允许在推理期间动态调整计算工作量，同时保持有竞争力的性能。我们通过使用能够生成中间预测以及相关置信度得分的侧分支来增强深度Q网络（DQN）来实现这一点。我们还提出了一种新的训练方法，用于在动态RL设置中学习动作和分支置信度。我们的实验评估所提出的框架雅达利2600游戏任务和现实的无人机导航任务的开源无人机模拟器（PEOPLE）。我们表明，与没有任何侧分支的基线DQN相比，RAPID-RL在Atari（无人机导航）任务上保持0.88倍（0.91倍）以上的性能的同时，会产生0.34倍（0.25倍）的操作数（OPS）。OPS的减少导致快速和有效的推理，证明对于资源受限的边缘非常有益，其中以最小的计算做出快速决策是必不可少的。
英文摘要：
Present-day Deep Reinforcement Learning (RL) systems show great promise towards building intelligent agents surpassing human-level performance. However, the computational complexity associated with the underlying deep neural networks (DNNs) leads to power-hungry implementations. This makes deep RL systems unsuitable for deployment on resource-constrained edge devices. To address this challenge, we propose a reconfigurable architecture with preemptive exits for efficient deep RL (RAPID-RL). RAPID-RL enables conditional activation of DNN layers based on the difficulty level of inputs. This allows to dynamically adjust the compute effort during inference while maintaining competitive performance. We achieve this by augmenting a deep Q-network (DQN) with side-branches capable of generating intermediate predictions along with an associated confidence score. We also propose a novel training methodology for learning the actions and branch confidence scores in a dynamic RL setting. Our experiments evaluate the proposed framework for Atari 2600 gaming tasks and a realistic Drone navigation task on an open-source drone simulator (PEDRA). We show that RAPID-RL incurs 0.34x (0.25x) number of operations (OPS) while maintaining performance above 0.88x (0.91x) on Atari (Drone navigation) tasks, compared to a baseline-DQN without any side-branches. The reduction in OPS leads to fast and efficient inference, proving to be highly beneficial for the resource-constrained edge where making quick decisions with minimal compute is essential.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
