AMAGO：面向自适应智能体的可扩展上下文强化学习
AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents
AI总结：
本研究提出可扩展的上下文强化学习智能体AMAGO，通过重新设计离策略上下文方法实现长序列Transformer的并行端到端训练，在元RL、长期记忆及目标条件开放世界任务中表现优异。
摘要：
我们提出了AMAGO，这是一种上下文强化学习（RL）智能体，利用序列模型来应对泛化、长期记忆和元学习带来的挑战。近期研究表明，离策略学习可让基于循环策略的上下文强化学习具备可行性。但这类方法需要大量调优，且会在智能体的记忆容量、规划时域和模型规模上形成关键瓶颈，限制了可扩展性。AMAGO重新审视并重新设计了离策略上下文方法，能够结合端到端强化学习，在完整的轨迹 rollout 上并行训练长序列Transformer。我们的智能体具备可扩展性，适用于广泛问题，我们在元强化学习和长期记忆领域的实验验证了其优异性能。AMAGO聚焦稀疏奖励和离策略数据的特性，还让上下文学习能够拓展到探索难度高的目标条件问题中。结合多目标后见之明重标记方案后，AMAGO能够解决此前难以处理的一类开放世界领域问题——在程序化生成的环境中，智能体需要完成大量可能的指令。
英文摘要：
We introduce AMAGO, an in-context Reinforcement Learning (RL) agent that uses sequence models to tackle the challenges of generalization, long-term memory, and meta-learning. Recent works have shown that off-policy learning can make in-context RL with recurrent policies viable. Nonetheless, these approaches require extensive tuning and limit scalability by creating key bottlenecks in agents' memory capacity, planning horizon, and model size. AMAGO revisits and redesigns the off-policy in-context approach to successfully train long-sequence Transformers over entire rollouts in parallel with end-to-end RL. Our agent is scalable and applicable to a wide range of problems, and we demonstrate its strong performance empirically in meta-RL and long-term memory domains. AMAGO's focus on sparse rewards and off-policy data also allows in-context learning to extend to goal-conditioned problems with challenging exploration. When combined with a multi-goal hindsight relabeling scheme, AMAGO can solve a previously difficult category of open-world domains, where agents complete many possible instructions in procedurally generated environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
