arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.05844cs.LGcs.AIcs.SYeess.SY
收藏

具有通用策略的不确定性系统辨识

Uncertainty Aware System Identification with Universal Policies

Applied Artificial Intelligence Institute(应用人工智能研究所);Deakin University(迪肯大学)

Buddhika Laknath Semage, Thommen George Karimpanal, Santu Rana, Svetha Venkatesh

摘要:

Sim2real transfer主要涉及将在模拟中训练的策略转移到可能存在噪声的现实世界环境中。与sim2real transfer相关的一个常见问题是估计真实世界的环境参数,以使模拟环境接地。虽然现有的方法,如域随机化(DR)可以通过在训练过程中从参数的分布中采样来产生鲁棒的策略,但没有建立用于识别给定现实世界设置的相应分布的参数的方法。在这项工作中,我们提出了不确定性感知的策略搜索(UncAPS),我们使用通用策略网络(UPN)在整个环境参数范围内存储模拟训练的特定于任务的策略,然后采用强大的贝叶斯优化,通过以DR方式组合相关的UPN策略,为给定的环境制定强大的策略。这种政策驱动的基础预计将更有效,因为它只估计与任务相关的参数集。此外,我们还考虑了搜索过程中的估计不确定性,以产生对任意和认知不确定性都具有鲁棒性的政策。我们经验评估我们的方法在一系列嘈杂的,连续的控制环境中,并显示其改进的性能相比,竞争基线。

英文摘要:

Sim2real transfer is primarily concerned with transferring policies trained in simulation to potentially noisy real world environments. A common problem associated with sim2real transfer is estimating the real-world environmental parameters to ground the simulated environment to. Although existing methods such as Domain Randomisation (DR) can produce robust policies by sampling from a distribution of parameters during training, there is no established method for identifying the parameters of the corresponding distribution for a given real-world setting. In this work, we propose Uncertainty-aware policy search (UncAPS), where we use Universal Policy Network (UPN) to store simulation-trained task-specific policies across the full range of environmental parameters and then subsequently employ robust Bayesian optimisation to craft robust policies for the given environment by combining relevant UPN policies in a DR like fashion. Such policy-driven grounding is expected to be more efficient as it estimates only task-relevant sets of parameters. Further, we also account for the estimation uncertainties in the search process to produce policies that are robust against both aleatoric and epistemic uncertainties. We empirically evaluate our approach in a range of noisy, continuous control environments, and show its improved performance compared to competing baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑