序列压缩加速强化学习中的信用分配
Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning
AI总结：
本文提出 Chunked-TD 算法，通过学习到的世界模型对轨迹进行分块压缩，在必要时自举以平滑偏差-方差权衡，从而加速强化学习中的信用分配，且比传统 TD($λ$) 更快解决问题。
摘要：
强化学习中的时间信用分配因延迟和随机结果而具有挑战性。Monte Carlo 目标能够弥合动作与结果之间的长延迟，但由于随机性会导致高方差目标。时序差分（TD）学习使用自举来克服方差，但引入了只能通过多次迭代来纠正的偏差。TD($λ$) 提供了一种平滑导航这种偏差-方差权衡的机制。适当选择 $λ$ 可以显著提高性能。在这里，我们提出了 Chunked-TD，它使用模型预测的转移概率来计算 $λ$-回报目标。与其他基于模型的信用分配解决方案不同，Chunked-TD 较不易受模型不准确性的影响。我们的方法受到历史压缩原理的启发，并对传统 TD 学习的轨迹进行“分块”。使用学习到的世界模型进行分块，压缩了环境-策略交互的近确定性区域以加速信用分配，同时在必要时仍进行自举。我们提出了可以在线实现的算法，并表明它们比传统 TD($λ$) 更快地解决了一些问题。
英文摘要：
Temporal credit assignment in reinforcement learning is challenging due to delayed and stochastic outcomes. Monte Carlo targets can bridge long delays between action and consequence but lead to high-variance targets due to stochasticity. Temporal difference (TD) learning uses bootstrapping to overcome variance but introduces a bias that can only be corrected through many iterations. TD($λ$) provides a mechanism to navigate this bias-variance tradeoff smoothly. Appropriately selecting $λ$ can significantly improve performance. Here, we propose Chunked-TD, which uses predicted probabilities of transitions from a model for computing $λ$-return targets. Unlike other model-based solutions to credit assignment, Chunked-TD is less vulnerable to model inaccuracies. Our approach is motivated by the principle of history compression and 'chunks' trajectories for conventional TD learning. Chunking with learned world models compresses near-deterministic regions of the environment-policy interaction to speed up credit assignment while still bootstrapping when necessary. We propose algorithms that can be implemented online and show that they solve some problems much faster than conventional TD($λ$).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
