保证满意与有限遗憾:认知满意价值函数的分析
Guaranteed satisficing and finite regret: Analysis of a cognitive satisficing value function
AI总结:
针对强化学习复杂任务难在实际时间内解决的问题,提出风险敏感满意(RS)模型,在贪婪策略下整合风险态度实现满意策略,应用于K臂老虎机问题,证明其能找到达标动作且在特定期望水平下遗憾有限,并通过模拟验证性能。
摘要:
随着强化学习算法被应用于日益复杂和现实的任务,在实际时间框架内解决此类问题变得越来越困难。因此,我们关注一种满意策略,该策略寻找价值高于期望水平(类似于盈亏平衡点)的动作,而非最优动作。本文介绍了一个名为风险敏感满意(RS)的简单数学模型,它通过在贪婪策略下整合风险规避和风险倾向态度来实现满意策略。我们将所提模型应用于构成强化学习最基础任务类别的K臂老虎机问题,并证明了两个命题:一是RS保证能找到价值高于期望水平的动作;二是若将期望水平设为“最优水平”(即满意意味着最优),则RS的遗憾(期望损失)具有有限上界。我们通过数值模拟验证了结果,并将RS与K臂老虎机问题的其他代表性算法进行了性能比较。
英文摘要:
As reinforcement learning algorithms are being applied to increasingly complicated and realistic tasks, it is becoming increasingly difficult to solve such problems within a practical time frame. Hence, we focus on a \textit{satisficing} strategy that looks for an action whose value is above the aspiration level (analogous to the break-even point), rather than the optimal action. In this paper, we introduce a simple mathematical model called risk-sensitive satisficing ($RS$) that implements a satisficing strategy by integrating risk-averse and risk-prone attitudes under the greedy policy. We apply the proposed model to the $K$-armed bandit problems, which constitute the most basic class of reinforcement learning tasks, and prove two propositions. The first is that $RS$ is guaranteed to find an action whose value is above the aspiration level. The second is that the regret (expected loss) of $RS$ is upper bounded by a finite value, given that the aspiration level is set to an "optimal level" so that satisficing implies optimizing. We confirm the results through numerical simulations and compare the performance of $RS$ with that of other representative algorithms for the $K$-armed bandit problems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。