arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2207.08229cs.LGcs.ROstat.ML
收藏

用多步逆模型保证发现控制内隐态

Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Models

Microsoft Research(微软研究院);McGill University(麦吉尔大学);University of Montreal(蒙特利尔大学)

Alex Lamb, Riashat Islam, Yonathan Efroni, Aniket Didolkar, Dipendra Misra, Dylan Foster, Lekan Molu, Rajan Chari, Akshay Krishnamurthy, John Langford

摘要:

在许多顺序决策任务中,智能体无法对世界的全部复杂性进行建模,世界由大量相关和不相关的信息组成。例如,一个人走在一条城市街道上,试图对世界的各个方面进行建模,很快就会被大量的商店、汽车和进出视线的人所淹没,每一个人都遵循着自己复杂而神秘的动态。有没有可能把智能体的感觉信息变成一种最小的潜在状态,这种状态对于智能体在世界上成功地行动来说既必要又足够?我们制定了这个问题具体,并提出了代理控制内源状态发现算法(AC状态),它具有理论上的保证和实际证明,发现最小的控制内源性的潜在状态,其中包含控制代理所需的所有信息,同时完全丢弃所有不相关的信息。该算法由一个多步逆模型(从遥远的观察预测行动)与信息瓶颈。AC-State支持本地化、探索和导航,无需奖励或演示。我们在三个领域证明了控制内源性潜伏状态的发现:定位具有分心的机器人手臂(例如,改变照明条件和背景),与其他代理一起探索迷宫,并在Matterport House模拟器中导航。

英文摘要:

In many sequential decision-making tasks, the agent is not able to model the full complexity of the world, which consists of multitudes of relevant and irrelevant information. For example, a person walking along a city street who tries to model all aspects of the world would quickly be overwhelmed by a multitude of shops, cars, and people moving in and out of view, each following their own complex and inscrutable dynamics. Is it possible to turn the agent's firehose of sensory information into a minimal latent state that is both necessary and sufficient for an agent to successfully act in the world? We formulate this question concretely, and propose the Agent Control-Endogenous State Discovery algorithm (AC-State), which has theoretical guarantees and is practically demonstrated to discover the minimal control-endogenous latent state which contains all of the information necessary for controlling the agent, while fully discarding all irrelevant information. This algorithm consists of a multi-step inverse model (predicting actions from distant observations) with an information bottleneck. AC-State enables localization, exploration, and navigation without reward or demonstrations. We demonstrate the discovery of the control-endogenous latent state in three domains: localizing a robot arm with distractions (e.g., changing lighting conditions and background), exploring a maze alongside other agents, and navigating in the Matterport house simulator.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑