从像素到因子：学习用于强化学习的独立可控状态变量
From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learning
AI总结：
针对高维观测下强化学习无法利用因子化结构提升样本效率的问题，提出基于对比学习的动作可控因子化方法ACF，利用动作影响的稀疏性挖掘独立可控潜变量，在三个基准测试中性能优于现有解耦算法。
摘要：
利用因子化马尔可夫决策过程的算法比不依赖因子的方法样本效率高得多，但它们预先假设因子化表示已知——当智能体只能观测到高维输入时，这一要求就无法满足。相反，深度强化学习能处理这类输入，却无法从因子化结构中获益。我们提出动作可控因子化（Action-Controllable Factorization，ACF）方法来解决这一表示问题，这是一种对比学习方法，可挖掘独立可控的潜变量——即每个动作能分别影响的状态分量。ACF利用稀疏性：动作通常仅影响部分变量，其余变量则按环境动力学演化，为对比训练提供有效数据。在Taxi、FourRooms和MiniGrid-DoorKey三个已知因子化结构的基准测试中，ACF直接从像素观测中还原了真实可控因子，性能始终优于基线解耦算法。
英文摘要：
Algorithms that exploit factored Markov decision processes are far more sample-efficient than factor-agnostic methods, yet they assume a factored representation is known a priori -- a requirement that breaks down when the agent sees only high-dimensional observations. Conversely, deep reinforcement learning handles such inputs but cannot benefit from factored structure. We address this representation problem with Action-Controllable Factorization (ACF), a contrastive learning approach that uncovers independently controllable latent variables -- state components each action can influence separately. ACF leverages sparsity: actions typically affect only a subset of variables, while the rest evolve under the environment's dynamics, yielding informative data for contrastive training. ACF recovers the ground truth controllable factors directly from pixel observations on three benchmarks with known factored structure -- Taxi, FourRooms, and MiniGrid-DoorKey -- consistently outperforming baseline disentanglement algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
