arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.14552cs.LGcs.AI
收藏

将用于表示学习的多步逆模型推广到有限记忆 POMDPs

Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPs

Microsoft Research(微软研究院);New York University(纽约大学);DreamFold;MILA;Meta

Lili Wu, Ben Evans, Riashat Islam, Raihan Seraj, Yonathan Efroni, Alex Lamb

AI总结:

研究高维非马尔可夫环境下发现以智能体为中心的状态表示问题,提出将广义逆模型适应到有限记忆POMDPs中,并通过理论与实证分析验证其有效性及过去动作的双刃剑效应。

摘要:

发现一种信息丰富或以智能体为中心的状态表示(仅编码相关信息并丢弃无关信息)是扩展强化学习算法并将其高效应用于下游任务的关键挑战。先前的工作在高维马尔可夫环境中研究了这个问题,即当前观测可能是一个复杂对象但足以解码出信息状态。在本工作中,我们考虑在更具挑战性的高维非马尔可夫环境中发现以智能体为中心的状态的问题,此时状态可以从过去的观测序列中解码。我们证明广义逆模型可以适应于学习该任务的以智能体为中心的状态表示。我们的结果包括确定性动力学环境下的渐近理论,以及替代直觉算法的反例。我们通过对我们提出的不同替代方案在以智能体为中心的状态发现能力上的彻底实证研究来补充这些发现。特别值得注意的是我们对过去动作的分析,我们表明它们可能是一把双刃剑:正确使用时使算法更成功,错误使用时导致严重失败。

英文摘要:

Discovering an informative, or agent-centric, state representation that encodes only the relevant information while discarding the irrelevant is a key challenge towards scaling reinforcement learning algorithms and efficiently applying them to downstream tasks. Prior works studied this problem in high-dimensional Markovian environments, when the current observation may be a complex object but is sufficient to decode the informative state. In this work, we consider the problem of discovering the agent-centric state in the more challenging high-dimensional non-Markovian setting, when the state can be decoded from a sequence of past observations. We establish that generalized inverse models can be adapted for learning agent-centric state representation for this task. Our results include asymptotic theory in the deterministic dynamics setting as well as counter-examples for alternative intuitive algorithms. We complement these findings with a thorough empirical study on the agent-centric state discovery abilities of the different alternatives we put forward. Particularly notable is our analysis of past actions, where we show that these can be a double-edged sword: making the algorithms more successful when used correctly and causing dramatic failure when used incorrectly.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑