在高度模块化的强化学习库中集成分布式架构
Integrating Distributed Architectures in Highly Modular RL Libraries
AI总结：
针对强化学习库中模块化与分布式扩展难以结合的问题，提出通过独立可复用组件定义不同规模RL智能体的通用方案，可复现基准、适配多类分布式架构并解决复杂环境任务，为下一代RL库提供参考。
摘要：
推进强化学习（RL）发展需要足够灵活的工具，既能轻松实现新方法的原型设计，又能避免实验周转时间慢得不符合实际的问题。为满足第一项需求，最主流的RL库都倡导高度模块化的智能体可组合性，这为实验和开发提供了便利。为在合理时间范围内解决具有挑战性的环境任务，将RL扩展到大规模采样与计算资源已被证明是一种成功策略。但迄今为止，这种能力很难与模块化特性相结合。在本研究中，我们探索了支持本地与分布式执行层级下智能体可组合性的设计方案。我们提出了一种通用方法，可通过独立的可复用组件定义不同规模的RL智能体。实验证明，我们的设计方案能够复现经典基准（benchmark）、探索多种分布式架构，并解决全新的复杂环境任务，同时让用户能完全掌控智能体定义与训练方案的定义。我们相信本研究能为下一代RL库提供有益的参考思路。
英文摘要：
Advancing reinforcement learning (RL) requires tools that are flexible enough to easily prototype new methods while avoiding impractically slow experimental turnaround times. To match the first requirement, the most popular RL libraries advocate for highly modular agent composability, which facilitates experimentation and development. To solve challenging environments within reasonable time frames, scaling RL to large sampling and computing resources has proved a successful strategy. However, this capability has been so far difficult to combine with modularity. In this work, we explore design choices to allow agent composability both at a local and distributed level of execution. We propose a versatile approach that allows the definition of RL agents at different scales through independent reusable components. We demonstrate experimentally that our design choices allow us to reproduce classical benchmarks, explore multiple distributed architectures, and solve novel and complex environments while giving full control to the user in the agent definition and training scheme definition. We believe this work can provide useful insights to the next generation of RL libraries.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
