坏习惯：强化学习中的策略混淆与轨迹外泛化
Bad Habits: Policy Confounding and Out-of-Trajectory Generalization in RL
AI总结：
本文针对强化学习中智能体因策略收敛形成依赖特定轨迹的虚假关联习惯、偏离轨迹时表现失效的策略混淆问题，给出了该现象的数学刻画并通过示例说明其发生条件。
摘要：
强化学习智能体往往会养成仅在特定策略下有效的习惯。在智能体尝试不同动作的初始探索阶段之后，它们最终会收敛到某一特定策略。在这一过程中，状态-动作轨迹的分布会变窄，导致智能体反复经历相同的转移。这种重复暴露会催生特定观测与奖励之间的虚假相关性。智能体随后可能会捕捉到这些相关性，并养成适配其策略所规定的特定轨迹集合的简单化习惯。问题在于，当环境变化迫使智能体偏离其典型轨迹时，这些习惯可能会产生错误的结果。本文对这一被称为策略混淆（policy confounding）的现象进行了数学刻画，并通过一系列示例说明了其发生的条件。
英文摘要：
Reinforcement learning agents tend to develop habits that are effective only under specific policies. Following an initial exploration phase where agents try out different actions, they eventually converge onto a particular policy. As this occurs, the distribution over state-action trajectories becomes narrower, leading agents to repeatedly experience the same transitions. This repetitive exposure fosters spurious correlations between certain observations and rewards. Agents may then pick up on these correlations and develop simplistic habits tailored to the specific set of trajectories dictated by their policy. The problem is that these habits may yield incorrect outcomes when agents are forced to deviate from their typical trajectories, prompted by changes in the environment. This paper presents a mathematical characterization of this phenomenon, termed policy confounding, and illustrates, through a series of examples, the circumstances under which it occurs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
