arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1812.05795cs.AI
收藏

保证满意与有限遗憾:认知满意价值函数的分析

Guaranteed satisficing and finite regret: Analysis of a cognitive satisficing value function

Graduate School of Advanced Science and Engineering, Tokyo Denki University(东京电机大学先进科学与工程学院);School of Science and Engineering, Tokyo Denki University(东京电机大学理工学院);Dwango Artificial Intelligence Laboratory(Dwango人工智能实验室)

Akihiro Tamatsukuri, Tatsuji Takahashi

AI总结:

针对强化学习复杂任务难在实际时间内解决的问题,提出风险敏感满意(RS)模型,在贪婪策略下整合风险态度实现满意策略,应用于K臂老虎机问题,证明其能找到达标动作且在特定期望水平下遗憾有限,并通过模拟验证性能。

摘要:

随着强化学习算法被应用于日益复杂和现实的任务,在实际时间框架内解决此类问题变得越来越困难。因此,我们关注一种满意策略,该策略寻找价值高于期望水平(类似于盈亏平衡点)的动作,而非最优动作。本文介绍了一个名为风险敏感满意(RS)的简单数学模型,它通过在贪婪策略下整合风险规避和风险倾向态度来实现满意策略。我们将所提模型应用于构成强化学习最基础任务类别的K臂老虎机问题,并证明了两个命题:一是RS保证能找到价值高于期望水平的动作;二是若将期望水平设为“最优水平”(即满意意味着最优),则RS的遗憾(期望损失)具有有限上界。我们通过数值模拟验证了结果,并将RS与K臂老虎机问题的其他代表性算法进行了性能比较。

英文摘要:

As reinforcement learning algorithms are being applied to increasingly complicated and realistic tasks, it is becoming increasingly difficult to solve such problems within a practical time frame. Hence, we focus on a \textit{satisficing} strategy that looks for an action whose value is above the aspiration level (analogous to the break-even point), rather than the optimal action. In this paper, we introduce a simple mathematical model called risk-sensitive satisficing ($RS$) that implements a satisficing strategy by integrating risk-averse and risk-prone attitudes under the greedy policy. We apply the proposed model to the $K$-armed bandit problems, which constitute the most basic class of reinforcement learning tasks, and prove two propositions. The first is that $RS$ is guaranteed to find an action whose value is above the aspiration level. The second is that the regret (expected loss) of $RS$ is upper bounded by a finite value, given that the aspiration level is set to an "optimal level" so that satisficing implies optimizing. We confirm the results through numerical simulations and compare the performance of $RS$ with that of other representative algorithms for the $K$-armed bandit problems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑