通过策略多样性进行技能学习可为强化学习产生可辨识的表征
Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learning
AI总结：
本文从可辨识表征学习角度研究互信息技能学习，证明对比后继特征方法可在线性变换意义下可证明地恢复环境真实特征，并在 MuJoCo 与 DeepMind Control 上验证了该结论。
摘要：
强化学习（RL）中的自监督特征学习与预训练方法通常依赖信息论原理，被称为互信息技能学习（MISL）。这些方法旨在学习环境表征，同时激励对环境的探索。然而，MISL 中表征与互信息参数化的作用在理论上尚未得到充分理解。我们的工作通过可辨识表征学习的视角，聚焦于对比后继特征（CSF）方法来研究 MISL。我们证明，由于特征的内积参数化以及判别意义上的技能多样性，CSF 可以可证明地恢复环境的真实特征，至多相差一个线性变换。这一针对强化学习中表征学习的首个可辨识性保证，也有助于解释不同互信息目标的含义以及熵正则化项的缺点。我们在 MuJoCo 和 DeepMind Control 上通过实验验证了我们的主张，并展示了 CSF 如何可证明地从状态和像素中恢复真实特征。
英文摘要：
Self-supervised feature learning and pretraining methods in reinforcement learning (RL) often rely on information-theoretic principles, termed mutual information skill learning (MISL). These methods aim to learn a representation of the environment while also incentivizing exploration thereof. However, the role of the representation and mutual information parametrization in MISL is not yet well understood theoretically. Our work investigates MISL through the lens of identifiable representation learning by focusing on the Contrastive Successor Features (CSF) method. We prove that CSF can provably recover the environment's ground-truth features up to a linear transformation due to the inner product parametrization of the features and skill diversity in a discriminative sense. This first identifiability guarantee for representation learning in RL also helps explain the implications of different mutual information objectives and the downsides of entropy regularizers. We empirically validate our claims in MuJoCo and DeepMind Control and show how CSF provably recovers the ground-truth features both from states and pixels.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
