面向感知强化学习的差分编码观测空间
Differentially Encoded Observation Spaces for Perceptive Reinforcement Learning
AI总结:
针对感知深度强化学习训练内存消耗大的问题,提出利用无损差分视频编码压缩经验回放缓冲区的方法,可大幅降低内存占用并提升任务运行延迟。
摘要:
感知深度强化学习(DRL)近年来依托基于图像的输入数据,在复杂AI系统领域取得了诸多突破,其应用成果涵盖了达到超人类水平的电子游戏智能体,以及具备灵巧操作能力的物理智能机器人。然而,训练这类搭载感知DRL的系统目前仍需消耗极高的计算资源与内存资源,往往需要庞大的训练数据集与大容量的经验回放缓冲区,这对下一代野外机器人构成了挑战——这类机器人需要具备边缘侧学习能力才能适应所处环境。本文提出通过差分编码观测空间来解决这一问题:我们将存储的基于图像的观测值重新解读为视频,利用无损差分视频编码方案对回放缓冲区进行压缩,且不会影响训练性能。我们采用3种当前最先进的DRL算法对所提方法进行评估,结果显示,在Atari 2600基准测试和DeepMind Control Suite(DMC)的各类任务中,差分图像编码分别将内存占用降低了至多14.2倍和16.7倍。这些内存节省还使得此前需要在闪存与RAM之间分页调度的大规模感知DRL可完全在RAM中运行,将DMC任务的延迟降低了至多32%。
英文摘要:
Perceptive deep reinforcement learning (DRL) has lead to many recent breakthroughs for complex AI systems leveraging image-based input data. Applications of these results range from super-human level video game agents to dexterous, physically intelligent robots. However, training these perceptive DRL-enabled systems remains incredibly compute and memory intensive, often requiring huge training datasets and large experience replay buffers. This poses a challenge for the next generation of field robots that will need to be able to learn on the edge in order to adapt to their environments. In this paper, we begin to address this issue through differentially encoded observation spaces. By reinterpreting stored image-based observations as a video, we leverage lossless differential video encoding schemes to compress the replay buffer without impacting training performance. We evaluate our approach with three state-of-the-art DRL algorithms and find that differential image encoding reduces the memory footprint by as much as 14.2x and 16.7x across tasks from the Atari 2600 benchmark and the DeepMind Control Suite (DMC) respectively. These savings also enable large-scale perceptive DRL that previously required paging between flash and RAM to be run entirely in RAM, improving the latency of DMC tasks by as much as 32%.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。