强化学习中的自确认叠加陷阱
Self-Confirming Superposition Traps in Reinforcement Learning
浏览论文内容
中文总结 AI 辅助
本研究揭示强化学习中一种自确认叠加陷阱：表示拟合的最优性可能维持低回报策略，并提出保留被忽略状态访问等干预措施以改善控制性能。
中文摘要 AI 辅助
强化学习（RL）在由智能体策略选择的数据上训练表示，然后利用由此产生的回报来指导其下一步选择。我们证明，即使在这些数据上表示拟合是全局最优的，这个循环也能维持一个较低回报的策略。在自确认叠加陷阱中，每个最优编码都将重叠的方向分配给在当前策略下很少同时出现的特征。另一种行动使它们同时出现，导致干扰，从而降低其回报并强化回避行为，尽管在相同容量下重新拟合该行动会产生更多回报。我们在一个捆绑的两步模型中刻画了允许陷阱存在的维度，并分别表明相等的特征频率、持续访问以及独立的控制器学习不一定能防止陷阱。由于拟合按访问频率对误差加权，一个被回避的行动可能以很小的目标代价失去其回报优势。在一个有限行动模型中，我们界定了这种扭曲，并推导出一个重放条件：对最佳单独适应行动给予足够的训练权重，即使在存在残余误差的情况下也能保持其排名。神经PPO实验展示了反馈在学习过程中如何发展：初始化为不同行动的智能体会产生不同的干扰模式、相反的均值回报排名，以及在相同容量下的不同最终策略。因此，我们测试保留对被忽略状态的访问是否能改善控制。在训练中保留这些状态可减少测得的干扰并提高顺序回报，即使在编码器冻结时也能获得收益。相关的状态访问、重放权重和特征重叠干预措施改善了MiniGrid和DMControl上的控制。对于通过世界模型学习的智能体，受保护的拟合在容量不变的情况下提高了DreamerV3--Crafter的累积训练分数。
英文摘要
Reinforcement learning (RL) trains representations on data selected by the agent's policy, which then uses the resulting returns to guide its next choices. We show that this loop can sustain a lower-return policy even when representation fitting is globally optimal on those data. In a self-confirming superposition trap, every optimal code assigns overlapping directions to features that rarely occur together under the current policy. An alternative action brings them together, causing interference that lowers its return and reinforces avoidance, although refitting to that action would yield more return at the same capacity. We characterize the dimensions admitting a trap in a tied two-step model and show separately that equal feature frequencies, continued visitation, and independent controller learning need not prevent it. Because fitting weights errors by visitation, an avoided action can lose its return advantage at little cost to the objective. In a finite-action model, we bound this distortion and derive a replay condition: sufficient training weight on the best separately adapted action preserves its ranking despite residual error. Neural PPO experiments show how the feedback develops during learning: agents initialized toward different actions develop different interference patterns, opposite mean return rankings, and different final policies at the same capacity. We therefore test whether retaining access to neglected states can improve control. Keeping these states in training reduces measured interference and improves sequential return, with gains even when the encoder is frozen. Related interventions on state access, replay weights, and feature overlap improve control on MiniGrid and DMControl. For agents that learn through a world model, protected fitting improves DreamerV3--Crafter's cumulative training scores at unchanged capacity.
