Sim2Real学习的权衡：真实世界的学习比模拟更快
Trade-off on Sim2Real Learning: Real-world Learning Faster than Simulations
摘要：
由于深度神经网络需要大量的训练样本，因此深度强化学习（DRL）实验通常在模拟环境中进行。相比之下，基于模型的贝叶斯学习允许机器人在现实世界中的几次试验中学习好的策略。虽然贝叶斯方法需要较少的迭代次数，但每次试验的计算成本相对较高，并且这种方法的优势与维数和噪声密切相关。在这里，我们将深度贝叶斯学习算法与无模型DRL算法进行比较，同时分析我们从模拟和真实世界实验中收集的结果。在考虑Sim和Real学习时，我们的实验表明，即使考虑到计算时间（而不是迭代次数），样本高效的Deep Bayesian RL性能也优于DRL。此外，在模拟和实验中执行的深度贝叶斯RL之间的计算时间差异指向了一条穿越现实差距的可行路径。我们还表明，Sim和Real之间的混合并不优于纯粹的Real方法，指出现实可以为贝叶斯学习提供最佳先验知识的可能性。机器人专家每天都在设计和制造机器人，我们的研究结果表明，在现实世界中更高的学习效率将通过跳过模拟来缩短设计和部署之间的时间。
英文摘要：
Deep Reinforcement Learning (DRL) experiments are commonly performed in simulated environments due to the tremendous training sample demands from deep neural networks. In contrast, model-based Bayesian Learning allows a robot to learn good policies within a few trials in the real world. Although it takes fewer iterations, Bayesian methods pay a relatively higher computational cost per trial, and the advantage of such methods is strongly tied to dimensionality and noise. In here, we compare a Deep Bayesian Learning algorithm with a model-free DRL algorithm while analyzing our results collected from both simulations and real-world experiments. While considering Sim and Real learning, our experiments show that the sample-efficient Deep Bayesian RL performance is better than DRL even when computation time (as opposed to number of iterations) is taken in consideration. Additionally, the difference in computation time between Deep Bayesian RL performed in simulation and in experiments point to a viable path to traverse the reality gap. We also show that a mix between Sim and Real does not outperform a purely Real approach, pointing to the possibility that reality can provide the best prior knowledge to a Bayesian Learning. Roboticists design and build robots every day, and our results show that a higher learning efficiency in the real-world will shorten the time between design and deployment by skipping simulations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
