赌徒的问题和超越
The Gambler's Problem and Beyond
摘要：
我们分析了赌徒问题，这是一个简单的强化学习问题，赌徒有机会加倍或输掉赌注，直到达到目标。这是Sutton和Barto（2018）在强化学习教科书中介绍的一个早期例子，他们提到了一个有趣的最优值函数模式，其中包含高频分量和重复的非光滑点。然而，没有进一步调查。我们提供了离散和连续情况下的最优值函数的精确公式。虽然看起来很简单，但价值函数是病态的：分形、自相似、取零或无穷大的导数，而且不能写成初等函数。事实上，它是广义康托函数之一，它拥有迄今为止未知的复杂性。我们的分析可以为在实际应用和实现中改进值函数近似、基于梯度的算法和Q学习提供见解。
英文摘要：
We analyze the Gambler's problem, a simple reinforcement learning problem where the gambler has the chance to double or lose the bets until the target is reached. This is an early example introduced in the reinforcement learning textbook by Sutton and Barto (2018), where they mention an interesting pattern of the optimal value function with high-frequency components and repeating non-smooth points. It is however without further investigation. We provide the exact formula for the optimal value function for both the discrete and the continuous cases. Though simple as it might seem, the value function is pathological: fractal, self-similar, derivative taking either zero or infinity, and not written as elementary functions. It is in fact one of the generalized Cantor functions, where it holds a complexity that has been uncharted thus far. Our analyses could provide insights into improving value function approximation, gradient-based algorithms, and Q-learning, in real applications and implementations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
