An important step in the design of autonomous systems is to evaluate the probability that a failure will occur. In safety-critical domains, the failure probability is extremely small so that the evaluation of a policy through Monte Carlo sampling is inefficient. Adaptive importance sampling approaches have been developed for rare event estimation but do not scale well to sequential systems with long horizons. In this work, we develop two adaptive importance sampling algorithms that can efficiently estimate the probability of rare events for sequential decision making systems. The basis for these algorithms is the minimization of the Kullback-Leibler divergence between a state-dependent proposal distribution and a target distribution over trajectories, but the resulting algorithms resemble policy gradient and value-based reinforcement learning. We apply multiple importance sampling to reduce the variance of our estimate and to address the issue of multi-modality in the optimal proposal distribution. We demonstrate our approach on a control task with both continuous and discrete actions spaces and show accuracy improvements over several baselines.
University of California, Berkeley(加州大学伯克利分校);University of Nottingham(诺丁汉大学);Kavli Energy NanoScience Institute(卡夫利能源纳米科学研究所);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室)
Behavior of prediction performance metrics with rare events
Department of Biostatistics, University of Washington(华盛顿大学生物统计学系);Biostatistics Division, Kaiser Permanente Washington Health Research Institute(凯撒医疗集团华盛顿健康研究中心生物统计学部);Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Center(弗雷德 Hutchinson癌症中心疫苗与传染病部)