arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.21186cs.LGcs.AIcs.RO
收藏

通过学习的时间抽象实现随机环境中的可扩展决策

Scalable Decision-Making in Stochastic Environments through Learned Temporal Abstraction

Vanderbilt University(范德堡大学);Pennsylvania State University(宾夕法尼亚州立大学);Nissan Advanced Technology Center(日产先进技术中心)

Baiting Luo, Ava Pettet, Aron Laszka, Abhishek Dubey, Ayan Mukhopadhyay

AI总结:

提出 Latent Macro Action Planner (L-MAP),通过 VQ-VAE 学习时间扩展宏动作以降低动作维度,并结合 MCTS 处理环境与策略的随机性,在高维随机环境的离线 RL 中实现了低延迟且高效的规划决策。

摘要:

高维连续动作空间中的序贯决策,特别是在随机环境中,面临着重大的计算挑战。我们在传统的离线RL设置中探索这一挑战,其中智能体必须基于通过随机行为策略收集的数据来学习如何做出决策。我们提出了 Latent Macro Action Planner (L-MAP),它通过状态条件的 Vector Quantized Variational Autoencoder (VQ-VAE) 学习一组时间扩展的宏动作,从而有效降低动作维度,以解决这一挑战。L-MAP 采用一个(独立的)学习先验模型,该模型充当潜在转移模型,并允许对合理动作进行高效采样。在规划过程中,我们的方法通过使用 Monte Carlo tree search (MCTS) 来考虑环境和行为策略中的随机性。在离线RL设置中,包括随机连续控制任务,L-MAP 在离散潜在动作上高效搜索以产生高期望回报。实证结果表明,尽管动作维度增加,L-MAP 仍保持低决策延迟。值得注意的是,在从具有固有随机动力学的连续控制到高维机器人手部操作的任务中,L-MAP 显著优于现有的基于模型的方法,并与强大的无模型 actor-critic 基线表现相当,突显了所提出方法在具有高维动作空间的复杂和随机环境中进行规划的有效性。

英文摘要:

Sequential decision-making in high-dimensional continuous action spaces, particularly in stochastic environments, faces significant computational challenges. We explore this challenge in the traditional offline RL setting, where an agent must learn how to make decisions based on data collected through a stochastic behavior policy. We present Latent Macro Action Planner (L-MAP), which addresses this challenge by learning a set of temporally extended macro-actions through a state-conditional Vector Quantized Variational Autoencoder (VQ-VAE), effectively reducing action dimensionality. L-MAP employs a (separate) learned prior model that acts as a latent transition model and allows efficient sampling of plausible actions. During planning, our approach accounts for stochasticity in both the environment and the behavior policy by using Monte Carlo tree search (MCTS). In offline RL settings, including stochastic continuous control tasks, L-MAP efficiently searches over discrete latent actions to yield high expected returns. Empirical results demonstrate that L-MAP maintains low decision latency despite increased action dimensionality. Notably, across tasks ranging from continuous control with inherently stochastic dynamics to high-dimensional robotic hand manipulation, L-MAP significantly outperforms existing model-based methods and performs on-par with strong model-free actor-critic baselines, highlighting the effectiveness of the proposed approach in planning in complex and stochastic environments with high-dimensional action spaces.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑