分析近似值迭代算法
Analyzing Approximate Value Iteration Algorithms
AI总结：
本文研究了在Bellman算子的近似值存在噪声和可能偏差的情况下，值迭代方案的随机迭代版本，即近似值迭代（AVI）方案。通过神经网络近似Bellman算子，考虑了训练样本带来的误差和偏差，并提出了可验证的充分条件，确保AVI在几乎必然有界的情况下收敛到近似Bellman算子的不动点。此外，还展示了AVI稳定性分析可以扩展到多值随机近似的一般情况，并可用于寻找收缩多值映射的不动点。
摘要：
在本文中，我们考虑了值迭代方案的随机迭代版本，其中仅能获得Bellman算子的噪声和可能有偏的近似值。我们将这种版本称为近似值迭代（AVI）方案。神经网络常用于函数逼近，以克服Bellman的维度诅咒。在本文中，它们被用于近似Bellman算子。由于神经网络通常通过样本数据进行训练，可能会引入误差和偏差。AVI的设计考虑了具有Bellman算子有偏近似值和采样误差的实现。我们提出了可验证的充分条件，确保AVI在几乎必然有界的情况下收敛到近似Bellman算子的不动点。为了确保AVI的稳定性，我们提出了三种不同但相关的充分条件集，这些条件基于适当Lyapunov函数的存在。这些基于Lyapunov函数的条件易于验证且是新的。可验证性还增强了由于提供了构造必要Lyapunov函数的食谱。我们还展示了AVI的稳定性分析可以轻易扩展到多值随机近似的普遍情况。最后，我们展示AVI也可用于更一般的情况，即寻找收缩多值映射的不动点。
英文摘要：
In this paper, we consider the stochastic iterative counterpart of the value iteration scheme wherein only noisy and possibly biased approximations of the Bellman operator are available. We call this counterpart as the approximate value iteration (AVI) scheme. Neural networks are often used as function approximators, in order to counter Bellman's curse of dimensionality. In this paper, they are used to approximate the Bellman operator. Since neural networks are typically trained using sample data, errors and biases may be introduced. The design of AVI accounts for implementations with biased approximations of the Bellman operator and sampling errors. We present verifiable sufficient conditions under which AVI is stable (almost surely bounded) and converges to a fixed point of the approximate Bellman operator. To ensure the stability of AVI, we present three different yet related sets of sufficient conditions that are based on the existence of an appropriate Lyapunov function. These Lyapunov function based conditions are easily verifiable and new to the literature. The verifiability is enhanced by the fact that a recipe for the construction of the necessary Lyapunov function is also provided. We also show that the stability analysis of AVI can be readily extended to the general case of set-valued stochastic approximations. Finally, we show that AVI can also be used in more general circumstances, i.e., for finding fixed points of contractive set-valued maps.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
