模型容量有限条件下的动态规划
Selective Dyna-style Planning Under Limited Model Capacity
摘要：
在基于模型的强化学习中，使用不完美的环境模型进行规划可能会损害学习进度。但是，即使模型是不完美的，它仍然可能包含对规划有用的信息。在本文中，我们研究的想法，选择性地使用一个不完美的模型。智能体应该在状态空间中模型有用的部分进行规划，但避免在模型有害的部分使用模型。一个有效的选择性规划机制需要估计预测的不确定性，这是由任意的不确定性，参数的不确定性，模型的不充分性，以及其他来源。先前的工作集中在选择性规划的参数不确定性。在这项工作中，我们强调模型不足的重要性。我们发现，异方差回归可以信号预测的不确定性所产生的模型不足，这是互补的，这是检测的方法设计的参数不确定性，这表明，考虑参数的不确定性和模型不足可能是一个更有前途的方向，有效的选择性规划比任何一个孤立的。
英文摘要：
In model-based reinforcement learning, planning with an imperfect model of the environment has the potential to harm learning progress. But even when a model is imperfect, it may still contain information that is useful for planning. In this paper, we investigate the idea of using an imperfect model selectively. The agent should plan in parts of the state space where the model would be helpful but refrain from using the model where it would be harmful. An effective selective planning mechanism requires estimating predictive uncertainty, which arises out of aleatoric uncertainty, parameter uncertainty, and model inadequacy, among other sources. Prior work has focused on parameter uncertainty for selective planning. In this work, we emphasize the importance of model inadequacy. We show that heteroscedastic regression can signal predictive uncertainty arising from model inadequacy that is complementary to that which is detected by methods designed for parameter uncertainty, indicating that considering both parameter uncertainty and model inadequacy may be a more promising direction for effective selective planning than either in isolation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
