arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.11940cs.LGcs.SYeess.SY
收藏

多步逆推并非你所需的一切

Multistep Inverse Is Not All You Need

The University of Texas at Austin(德克萨斯大学奥斯汀分校);Sony AI(索尼人工智能)

Alexander Levine, Peter Stone, Amy Zhang

AI总结:

针对 Ex-BMDP 中 AC-State 多步逆推表示学习失效的问题,本文提出结合潜在前向模型的 ACDF,并在表格与高维环境中验证其可正确恢复可控潜在状态。

摘要:

在现实世界的控制场景中,观测空间往往具有不必要的高维性,并且受到时间相关噪声的影响。然而,系统的可控动态通常远比原始观测的动态简单。因此,人们希望学习一个编码器,将观测空间映射到一个由控制相关变量构成的更简单空间。在本文中,我们研究 Ex-BMDP 模型;该模型最初由 Efroni 等人(2022)提出,用于形式化一类控制问题,其中观测可以分解为一个以确定性方式演化、依赖动作的潜在状态,以及与动作无关的时间相关噪声。Lamb 等人(2022)提出“AC-State”方法,用于在此类问题中学习编码器,以从观测中提取完整的、依赖动作的潜在状态表示。AC-State 是一种多步逆推方法,因为它使用一条路径中首个状态和最后一个状态的编码来预测该路径中的第一个动作。然而,我们识别出若干情形,在这些情形下 AC-State 无法学到关于状态中智能体可控因素的正确潜在表示。因此,我们提出一种新算法 ACDF,将多步逆推预测与潜在前向模型相结合。ACDF 得到保证,能够为一大类 Ex-BMDP 模型正确推断出依赖动作的潜在状态编码器。我们通过数值模拟在表格型 Ex-BMDP 上证明了 ACDF 的有效性;同时,也在使用基于神经网络的编码器的高维环境中证明了其有效性。代码可在 https://github.com/midi-lab/acdf 获取。

英文摘要:

In real-world control settings, the observation space is often unnecessarily high-dimensional and subject to time-correlated noise. However, the controllable dynamics of the system are often far simpler than the dynamics of the raw observations. It is therefore desirable to learn an encoder to map the observation space to a simpler space of control-relevant variables. In this work, we consider the Ex-BMDP model, first proposed by Efroni et al. (2022), which formalizes control problems where observations can be factorized into an action-dependent latent state which evolves deterministically, and action-independent time-correlated noise. Lamb et al. (2022) proposes the "AC-State" method for learning an encoder to extract a complete action-dependent latent state representation from the observations in such problems. AC-State is a multistep-inverse method, in that it uses the encoding of the the first and last state in a path to predict the first action in the path. However, we identify cases where AC-State will fail to learn a correct latent representation of the agent-controllable factor of the state. We therefore propose a new algorithm, ACDF, which combines multistep-inverse prediction with a latent forward model. ACDF is guaranteed to correctly infer an action-dependent latent state encoder for a large class of Ex-BMDP models. We demonstrate the effectiveness of ACDF on tabular Ex-BMDPs through numerical simulations; as well as high-dimensional environments using neural-network-based encoders. Code is available at https://github.com/midi-lab/acdf.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑