arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1312.0286cs.LGstat.ML
收藏

使用压缩预测状态的高效学习和规划

Efficient Learning and Planning with Compressed Predictive States

McGill University(麦吉尔大学)

William L. Hamilton, Mahdi Milani Fard, Joelle Pineau

摘要:

预测状态表示(PSR)为部分可观测系统建模提供了一个表达框架。通过将系统压缩表示为可观测量的函数,PSR学习方法避免使用局部最小值倾向的期望最大化,而是采用全局最优的基于矩的算法。此外,由于PSR不需要一个预定的潜在状态结构作为输入,他们提供了一个有吸引力的框架,基于模型的强化学习时,代理必须计划没有先验访问系统模型。遗憾的是,PSR的表达能力伴随着显著的计算成本,并且该成本是抑制PSR在应用中的使用的主要因素。为了克服这一缺点,我们引入了压缩PSR(CPSR)的概念。CPSR学习方法结合了降维、增量矩阵分解和压缩感知的最新进展。我们展示了这种方法如何为学习PSR的精确近似提供原则性途径,大大降低了与学习相关的计算成本,同时还提供了有效的正则化。更进一步,我们提出了一个规划框架,利用这些学习模型。我们表明,这种方法有利于模型学习和规划在大型复杂的部分可观察的领域,这是不可行的,如果没有压缩的原则使用的任务。

英文摘要:

Predictive state representations (PSRs) offer an expressive framework for modelling partially observable systems. By compactly representing systems as functions of observable quantities, the PSR learning approach avoids using local-minima prone expectation-maximization and instead employs a globally optimal moment-based algorithm. Moreover, since PSRs do not require a predetermined latent state structure as an input, they offer an attractive framework for model-based reinforcement learning when agents must plan without a priori access to a system model. Unfortunately, the expressiveness of PSRs comes with significant computational cost, and this cost is a major factor inhibiting the use of PSRs in applications. In order to alleviate this shortcoming, we introduce the notion of compressed PSRs (CPSRs). The CPSR learning approach combines recent advancements in dimensionality reduction, incremental matrix decomposition, and compressed sensing. We show how this approach provides a principled avenue for learning accurate approximations of PSRs, drastically reducing the computational costs associated with learning while also providing effective regularization. Going further, we propose a planning framework which exploits these learned models. And we show that this approach facilitates model-learning and planning in large complex partially observable domains, a task that is infeasible without the principled use of compression.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑