arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.12470cs.LGcs.AI
收藏

一种用于稀有事件估计的深度强化学习方法

A Deep Reinforcement Learning Approach to Rare Event Estimation

Stanford University(斯坦福大学)

Anthony Corso, Kyu-Young Kim, Shubh Gupta, Grace Gao, Mykel J. Kochenderfer

摘要:

自主系统设计中的一个重要步骤是评估故障发生的概率。在安全关键域中,失效概率非常小,因此通过蒙特卡罗抽样评估策略是低效的。自适应重要性采样方法已被开发用于稀有事件估计,但不能很好地扩展到具有长视野的连续系统。在这项工作中,我们开发了两个自适应重要性采样算法,可以有效地估计概率的顺序决策系统的罕见事件。这些算法的基础是最小化状态相关的建议分布和目标分布之间的Kullback-Leibler发散,但由此产生的算法类似于策略梯度和基于值的强化学习。我们应用多重重要性采样来减少我们估计的方差,并解决最优建议分布中的多模态问题。我们展示了我们的方法在控制任务与连续和离散的动作空间,并显示在几个基线的准确性提高。

英文摘要:

An important step in the design of autonomous systems is to evaluate the probability that a failure will occur. In safety-critical domains, the failure probability is extremely small so that the evaluation of a policy through Monte Carlo sampling is inefficient. Adaptive importance sampling approaches have been developed for rare event estimation but do not scale well to sequential systems with long horizons. In this work, we develop two adaptive importance sampling algorithms that can efficiently estimate the probability of rare events for sequential decision making systems. The basis for these algorithms is the minimization of the Kullback-Leibler divergence between a state-dependent proposal distribution and a target distribution over trajectories, but the resulting algorithms resemble policy gradient and value-based reinforcement learning. We apply multiple importance sampling to reduce the variance of our estimate and to address the issue of multi-modality in the optimal proposal distribution. We demonstrate our approach on a control task with both continuous and discrete actions spaces and show accuracy improvements over several baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑