基于信息价值策略的Ms. Pac-Man智能体专家能力分析
Analysis of Agent Expertise in Ms. Pac-Man using Value-of-Information-based Policies
摘要：
用于马尔可夫决策过程的传统强化学习方法依赖于弱引导的随机搜索来驱动学习过程。因此，很难预测可能出现的代理行为。在本文中，我们考虑一个信息理论的成本函数进行约束随机搜索，促进形成风险厌恶风险偏好的行为。这个成本函数是信息的价值，它提供了策略的预期回报和策略复杂度之间的最佳权衡;策略复杂度由比特数衡量，并由成本函数上的单个超参数控制。随着政策复杂性的降低，代理人将越来越多地避免冒险行为。这降低了高额累积奖励的可能性。随着政策复杂性的增加，代理人将采取行动，无论风险如何，都可以提高长期回报。可获得的奖励取决于一个单一的，可调的超参数，调节政策的复杂程度。我们评估的表现价值的信息为基础的政策上的随机版本的女士吃豆人。本文的一个主要组成部分是演示的范围内的政策复杂性值产生不同的游戏风格，并解释为什么会发生这种情况。我们还表明，我们的学习搜索机制比我们利用的其他更有效。这一结果意味着信息论的价值适用于强化学习中的开发-探索权衡。
英文摘要：
Conventional reinforcement learning methods for Markov decision processes rely on weakly-guided, stochastic searches to drive the learning process. It can therefore be difficult to predict what agent behaviors might emerge. In this paper, we consider an information-theoretic cost function for performing constrained stochastic searches that promote the formation of risk-averse to risk-favoring behaviors. This cost function is the value of information, which provides the optimal trade-off between the expected return of a policy and the policy's complexity; policy complexity is measured by number of bits and controlled by a single hyperparameter on the cost function. As the policy complexity is reduced, the agents will increasingly eschew risky actions. This reduces the potential for high accrued rewards. As the policy complexity increases, the agents will take actions, regardless of the risk, that can raise the long-term rewards. The obtainable reward depends on a single, tunable hyperparameter that regulates the degree of policy complexity. We evaluate the performance of value-of-information-based policies on a stochastic version of Ms. Pac-Man. A major component of this paper is the demonstration that ranges of policy complexity values yield different game-play styles and explaining why this occurs. We also show that our reinforcement-learning search mechanism is more efficient than the others we utilize. This result implies that the value of information theory is appropriate for framing the exploitation-exploration trade-off in reinforcement learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
