Transformer何时在RL中大放异彩？将记忆与信用分配解耦
When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment
AI总结：
该研究通过可配置任务将RL中的记忆长度与信用分配长度解耦，实证发现Transformer能增强长期记忆但不能改善长期信用分配。
摘要：
强化学习（RL）算法面临两个不同的挑战：学习对过去和当前观测的有效表征，以及确定动作如何影响未来回报。这两个挑战都涉及对长期依赖关系进行建模。Transformer架构在解决涉及长期依赖关系的问题方面非常成功，包括在RL领域。然而，基于Transformer的RL方法表现强劲的根本原因仍不清楚：是因为它们学习了有效的记忆，还是因为它们执行了有效的信用分配？在引入记忆长度和信用分配长度的形式化定义之后，我们设计了简单的可配置任务来测量这些不同的量。我们的实证结果表明，Transformer可以增强RL算法的记忆能力，可扩展到需要记忆1500步之前观测的任务。然而，Transformer并不会改善长期信用分配。总之，我们的结果为Transformer在RL中的成功提供了解释，同时也强调了未来研究和基准设计的一个重要领域。我们的代码已在https://github.com/twni2016/Memory-RL开源。
英文摘要：
Reinforcement learning (RL) algorithms face two distinct challenges: learning effective representations of past and present observations, and determining how actions influence future returns. Both challenges involve modeling long-term dependencies. The Transformer architecture has been very successful to solve problems that involve long-term dependencies, including in the RL domain. However, the underlying reason for the strong performance of Transformer-based RL methods remains unclear: is it because they learn effective memory, or because they perform effective credit assignment? After introducing formal definitions of memory length and credit assignment length, we design simple configurable tasks to measure these distinct quantities. Our empirical results reveal that Transformers can enhance the memory capability of RL algorithms, scaling up to tasks that require memorizing observations $1500$ steps ago. However, Transformers do not improve long-term credit assignment. In summary, our results provide an explanation for the success of Transformers in RL, while also highlighting an important area for future research and benchmark design. Our code is open-sourced at https://github.com/twni2016/Memory-RL
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
