Highway Value Iteration Networks

Factored Value Iteration Converges

In this paper we propose a novel algorithm, factored value iteration (FVI), for the approximate solution of factored Markov decision processes (fMDPs). The traditional approximate value iteration algorithm is modified in two ways. For one, the least-squares projection operator is modified so that it does not increase max-norm, and thus preserves convergence. The other modification is that we uniformly sample polynomially many samples from the (exponentially large) state space. This way, the complexity of our algorithm becomes polynomial in the size of the fMDP description length. We prove that…

带有门控摘要模块的值迭代网络

Value Iteration Networks with Gated Summarization Module

本文提出GS-VIN方法，通过自适应迭代策略减少迭代次数并提升稳定性，同时引入门控摘要模块强调整个规划过程，有效解决了VIN处理大地图及累积误差问题，显著提升了规划性能。

Generalized Value Iteration Networks: Life Beyond Lattices

In this paper, we introduce a generalized value iteration network (GVIN), which is an end-to-end neural network planning module. GVIN emulates the value iteration algorithm by using a novel graph convolution operator, which enables GVIN to learn and plan on irregular spatial graphs. We propose three novel differentiable kernels as graph convolution operators and show that the embedding based kernel achieves the best performance. We further propose episodic Q-learning, an improvement upon traditional n-step Q-learning that stabilizes training for networks that contain a planning module. Lastly,…

理解投影贝尔曼方程、线性Q学习和近似值迭代的理论性质

Understanding the theoretical properties of projected Bellman equation, linear Q-learning, and approximate value iteration

本文研究投影贝尔曼方程（PBE）及求解它的线性Q学习和近似值迭代（AVI）的理论性质，提出PBE解存在的两个充分条件，分析SNRDD假设对算法收敛的作用及与AVI收敛的关系，并给出ε-贪婪策略下PBE解的观察。

Geometric Policy Iteration for Markov Decision Processes

Recently discovered polyhedral structures of the value function for finite state-action discounted Markov decision processes (MDP) shed light on understanding the success of reinforcement learning. We investigate the value function polytope in greater detail and characterize the polytope boundary using a hyperplane arrangement. We further show that the value space is a union of finitely many cells of the same hyperplane arrangement and relate it to the polytope of the classical linear programming formulation for MDPs. Inspired by these geometric properties, we propose a new algorithm, Geometri…

Transfer Value Iteration Networks

Value iteration networks (VINs) have been demonstrated to have a good generalization ability for reinforcement learning tasks across similar domains. However, based on our experiments, a policy learned by VINs still fail to generalize well on the domain whose action space and feature space are not identical to those in the domain where it is trained. In this paper, we propose a transfer learning approach on top of VINs, termed Transfer VINs (TVINs), such that a learned policy from a source domain can be generalized to a target domain with only limited training data, even if the source domain a…

Value Iteration with Options and State Aggregation

This paper presents a way of solving Markov Decision Processes that combines state abstraction and temporal abstraction. Specifically, we combine state aggregation with the options framework and demonstrate that they work well together and indeed it is only after one combines the two that the full benefit of each is realized. We introduce a hierarchical value iteration algorithm where we first coarsely solve subgoals and then use these approximate solutions to exactly solve the MDP. This algorithm solved several problems faster than vanilla value iteration.

Graph neural induction of value iteration

Many reinforcement learning tasks can benefit from explicit planning based on an internal model of the environment. Previously, such planning components have been incorporated through a neural network that partially aligns with the computational graph of value iteration. Such network have so far been focused on restrictive environments (e.g. grid-worlds), and modelled the planning procedure only indirectly. We relax these constraints, proposing a graph neural network (GNN) that executes the value iteration (VI) algorithm, across arbitrary environment models, with direct supervision on the inte…

Deep Episodic Value Iteration for Model-based Meta-Reinforcement Learning

We present a new deep meta reinforcement learner, which we call Deep Episodic Value Iteration (DEVI). DEVI uses a deep neural network to learn a similarity metric for a non-parametric model-based reinforcement learning algorithm. Our model is trained end-to-end via back-propagation. Despite being trained using the model-free Q-learning objective, we show that DEVI's model-based internal structure provides `one-shot' transfer to changes in reward and transition structure, even for tasks with very high-dimensional state spaces.

连通 MDP 中值迭代的收敛性

On Value Iteration Convergence in Connected MDPs

针对连通 MDP，论文给出值迭代收敛的充分条件：唯一最优策略与遍历转移矩阵可保证折扣和平均回报准则下均获得快于 γ 的几何收敛速率。

Doubly-Asynchronous Value Iteration: Making Value Iteration Asynchronous in Actions

Value iteration (VI) is a foundational dynamic programming method, important for learning and planning in optimal control and reinforcement learning. VI proceeds in batches, where the update to the value of each state must be completed before the next batch of updates can begin. Completing a single batch is prohibitively expensive if the state space is large, rendering VI impractical for many applications. Asynchronous VI helps to address the large state space problem by updating one state at a time, in-place and in an arbitrary order. However, Asynchronous VI still requires a maximization ove…

XLVIN: eXecuted Latent Value Iteration Nets

Value Iteration Networks (VINs) have emerged as a popular method to incorporate planning algorithms within deep reinforcement learning, enabling performance improvements on tasks requiring long-range reasoning and understanding of environment dynamics. This came with several limitations, however: the model is not incentivised in any way to perform meaningful planning computations, the underlying state space is assumed to be discrete, and the Markov decision process (MDP) is assumed fixed and known. We propose eXecuted Latent Value Iteration Networks (XLVINs), which combine recent developments …

Generalized Second Order Value Iteration in Markov Decision Processes

Value iteration is a fixed point iteration technique utilized to obtain the optimal value function and policy in a discounted reward Markov Decision Process (MDP). Here, a contraction operator is constructed and applied repeatedly to arrive at the optimal solution. Value iteration is a first order method and therefore it may take a large number of iterations to converge to the optimal solution. Successive relaxation is a popular technique that can be applied to solve a fixed point equation. It has been shown in the literature that, under a special structure of the MDP, successive over-relaxati…

Value Iteration Networks

We introduce the value iteration network (VIN): a fully differentiable neural network with a `planning module' embedded within. VINs can learn to plan, and are suitable for predicting outcomes that involve planning-based reasoning, such as policies for reinforcement learning. Key to our approach is a novel differentiable approximation of the value-iteration algorithm, which can be represented as a convolutional neural network, and trained end-to-end using standard backpropagation. We evaluate VIN based policies on discrete and continuous path-planning domains, and on a natural-language based s…

Randomised Bayesian Least-Squares Policy Iteration

We introduce Bayesian least-squares policy iteration (BLSPI), an off-policy, model-free, policy iteration algorithm that uses the Bayesian least-squares temporal-difference (BLSTD) learning algorithm to evaluate policies. An online variant of BLSPI has been also proposed, called randomised BLSPI (RBLSPI), that improves its policy based on an incomplete policy evaluation step. In online setting, the exploration-exploitation dilemma should be addressed as we try to discover the optimal policy by using samples collected by ourselves. RBLSPI exploits the advantage of BLSTD to quantify our uncertai…

The Divergence of Reinforcement Learning Algorithms with Value-Iteration and Function Approximation

This paper gives specific divergence examples of value-iteration for several major Reinforcement Learning and Adaptive Dynamic Programming algorithms, when using a function approximator for the value function. These divergence examples differ from previous divergence examples in the literature, in that they are applicable for a greedy policy, i.e. in a "value iteration" scenario. Perhaps surprisingly, with a greedy policy, it is also possible to get divergence for the algorithms TD(1) and Sarsa(1). In addition to these divergences, we also achieve divergence for the Adaptive Dynamic Programmin…

通过绝对概率序列分析值迭代

Analysis of Value Iteration Through Absolute Probability Sequences

本文利用绝对概率序列建立新的分析框架，研究值迭代算法在 $L^2$ 范数下的收敛性，弥补以往主要关注无穷范数收敛分析的不足。

风险敏感指数代价马尔可夫决策过程中修正策略迭代的收敛性

On the Convergence of Modified Policy Iteration in Risk Sensitive Exponential Cost Markov Decision Processes

本文首次证明了修正策略迭代（MPI）在有限状态和动作空间的风险敏感指数代价马尔可夫决策过程中的收敛性，并仿真验证了其相较于策略迭代和价值迭代的计算效率优势。

Unifying Value Iteration, Advantage Learning, and Dynamic Policy Programming

Approximate dynamic programming algorithms, such as approximate value iteration, have been successfully applied to many complex reinforcement learning tasks, and a better approximate dynamic programming algorithm is expected to further extend the applicability of reinforcement learning to various tasks. In this paper we propose a new, robust dynamic programming algorithm that unifies value iteration, advantage learning, and dynamic policy programming. We call it generalized value iteration (GVI) and its approximated version, approximate GVI (AGVI). We show AGVI's performance guarantee, which i…

Improved Worst-Case Regret Bounds for Randomized Least-Squares Value Iteration

This paper studies regret minimization with randomized value functions in reinforcement learning. In tabular finite-horizon Markov Decision Processes, we introduce a clipping variant of one classical Thompson Sampling (TS)-like algorithm, randomized least-squares value iteration (RLSVI). Our $\tilde{\mathrm{O}}(H^2S\sqrt{AT})$ high-probability worst-case regret bound improves the previous sharpest worst-case regret bounds for RLSVI and matches the existing state-of-the-art worst-case TS-based regret bounds.

重新审视价值迭代：折扣奖励和平均奖励情况的统一分析

Revisiting Value Iteration: Unified Analysis of Discounted and Average-Reward Cases

本文通过统一几何分析，证明在特定条件下价值迭代在折扣和平均奖励设置中均具有更快的几何收敛速度。

Operator Splitting Value Iteration

We introduce new planning and reinforcement learning algorithms for discounted MDPs that utilize an approximate model of the environment to accelerate the convergence of the value function. Inspired by the splitting approach in numerical linear algebra, we introduce Operator Splitting Value Iteration (OS-VI) for both Policy Evaluation and Control problems. OS-VI achieves a much faster convergence rate when the model is accurate enough. We also introduce a sample-based version of the algorithm called OS-Dyna. Unlike the traditional Dyna architecture, OS-Dyna still converges to the correct value…

Multiagent Value Iteration Algorithms in Dynamic Programming and Reinforcement Learning

We consider infinite horizon dynamic programming problems, where the control at each stage consists of several distinct decisions, each one made by one of several agents. In an earlier work we introduced a policy iteration algorithm, where the policy improvement is done one-agent-at-a-time in a given order, with knowledge of the choices of the preceding agents in the order. As a result, the amount of computation for each policy improvement grows linearly with the number of agents, as opposed to exponentially for the standard all-agents-at-once method. For the case of a finite-state discounted …

Empirical Q-Value Iteration

We propose a new simple and natural algorithm for learning the optimal Q-value function of a discounted-cost Markov Decision Process (MDP) when the transition kernels are unknown. Unlike the classical learning algorithms for MDPs, such as Q-learning and actor-critic algorithms, this algorithm doesn't depend on a stochastic approximation-based method. We show that our algorithm, which we call the empirical Q-value iteration (EQVI) algorithm, converges to the optimal Q-value function. We also give a rate of convergence or a non-asymptotic sample complexity bound, and also show that an asynchrono…

随机控制问题的连续策略与值迭代及其收敛性

Continuous Policy and Value Iteration for Relaxed Stochastic Control Problems and Its Convergence

介绍连续策略-值迭代算法，用于随机控制问题，通过朗之万型动力学同时更新值函数和最优控制近似值，适用于两类控制问题，在哈密顿量单调性条件下收敛，可利用相关技术同时优化值函数与确定最优控制。

Value Iteration in Continuous Actions, States and Time

Classical value iteration approaches are not applicable to environments with continuous states and actions. For such environments, the states and actions are usually discretized, which leads to an exponential increase in computational complexity. In this paper, we propose continuous fitted value iteration (cFVI). This algorithm enables dynamic programming for continuous states and actions with a known dynamics model. Leveraging the continuous-time formulation, the optimal policy can be derived for non-linear control-affine dynamics. This closed-form solution enables the efficient extension of …

AsyncQVI: Asynchronous-Parallel Q-Value Iteration for Discounted Markov Decision Processes with Near-Optimal Sample Complexity

In this paper, we propose AsyncQVI, an asynchronous-parallel Q-value iteration for discounted Markov decision processes whose transition and reward can only be sampled through a generative model. Given such a problem with $|\mathcal{S}|$ states, $|\mathcal{A}|$ actions, and a discounted factor $γ\in(0,1)$, AsyncQVI uses memory of size $\mathcal{O}(|\mathcal{S}|)$ and returns an $\varepsilon$-optimal policy with probability at least $1-δ$ using $$\tilde{\mathcal{O}}\big(\frac{|\mathcal{S}||\mathcal{A}|}{(1-γ)^5\varepsilon^2}\log(\frac{1}δ)\big)$$ samples. AsyncQVI is also the first asynchronous…

Continuous-Time Fitted Value Iteration for Robust Policies

Solving the Hamilton-Jacobi-Bellman equation is important in many domains including control, robotics and economics. Especially for continuous control, solving this differential equation and its extension the Hamilton-Jacobi-Isaacs equation, is important as it yields the optimal policy that achieves the maximum reward on a give task. In the case of the Hamilton-Jacobi-Isaacs equation, which includes an adversary controlling the environment and minimizing the reward, the obtained policy is also robust to perturbations of the dynamics. In this paper we propose continuous fitted value iteration (…

一般和解博弈中Q值迭代的有限时间分析

Finite-Time Analysis of Q-Value Iteration for General-Sum Stackelberg Games

本文从控制理论视角研究了双玩家一般和解马尔可夫博弈中Stackelberg Q值迭代的收敛性，提出放松的策略条件并建模学习动态为切换系统，通过构造上下比较系统，建立了Q函数的有限时间误差界并刻画其收敛特性。

秩一修正值迭代

Rank-One Modified Value Iteration

本文提出秩一修正值迭代算法，用秩一近似和幂法处理马尔可夫决策过程的策略评估，理论上收敛速率与复杂度对标经典算法，实验中性能优于一阶算法及其加速版。

紧缩动力学值迭代

Deflated Dynamics Value Iteration

针对值迭代在折扣因子接近1时收敛慢的问题，提出利用矩阵分裂与紧缩移除转移矩阵主导特征结构的 DDVI，并扩展为 RL 中的 DDTD，显著改善收敛率。

基于理论指导的深度学习的国家道路网络长期交通建模

Interpretable long-term traffic modelling on national road networks using theory-informed deep learning

本文提出DeepDemand框架，结合交通理论与深度学习，通过社会经济特征和道路网络结构预测长期高速公路交通量，验证了其在长期规划中的有效性与地理可转移性。

Sublinear Least-Squares Value Iteration via Locality Sensitive Hashing

We present the first provable Least-Squares Value Iteration (LSVI) algorithms that have runtime complexity sublinear in the number of actions. We formulate the value function estimation procedure in value iteration as an approximate maximum inner product search problem and propose a locality sensitive hashing (LSH) [Indyk and Motwani STOC'98, Andoni and Razenshteyn STOC'15, Andoni, Laarhoven, Razenshteyn and Waingarten SODA'17] type data structure to solve this problem with sublinear time complexity. Moreover, we build the connections between the theory of approximate maximum inner product sea…

用于双因果最优传输的拟合值迭代方法

Fitted value iteration methods for bicausal optimal transport

本研究提出一种用于双因果最优传输的拟合值迭代方法，基于动态规划用函数类逼近值函数，证明了样本复杂度并验证神经网络满足假设，实验显示其在时间跨度增大时可扩展性优于现有方法且精度可接受。

Provably Efficient Reward-Agnostic Navigation with Linear Value Iteration

There has been growing progress on theoretical analyses for provably efficient learning in MDPs with linear function approximation, but much of the existing work has made strong assumptions to enable exploration by conventional exploration frameworks. Typically these assumptions are stronger than what is needed to find good solutions in the batch setting. In this work, we show how under a more standard notion of low inherent Bellman error, typically employed in least-square value iteration-style algorithms, we can provide strong PAC guarantees on learning a near optimal value function provided…

安全感知强化学习用于控制：通过风险敏感动作价值迭代和分位数回归

Safety-Aware Reinforcement Learning for Control via Risk-Sensitive Action-Value Iteration and Quantile Regression

本文提出一种基于风险敏感动作价值迭代和分位数回归的安全感知强化学习算法，通过整合条件价值在风险（CVaR）来确保安全，同时提供理论保证和实验验证，提升动态环境中的安全性能。

Off-Policy Interval Estimation with Lipschitz Value Iteration

Off-policy evaluation provides an essential tool for evaluating the effects of different policies or treatments using only observed data. When applied to high-stakes scenarios such as medical diagnosis or financial decision-making, it is crucial to provide provably correct upper and lower bounds of the expected reward, not just a classical single point estimate, to the end-users, as executing a poor policy can be very costly. In this work, we propose a provably correct method for obtaining interval bounds for off-policy evaluation in a general continuous setting. The idea is to search for the …

基于价值迭代的最优自适应控制稳定性分析

Stability Analysis of Optimal Adaptive Control using Value Iteration with Approximation Errors

本文通过价值迭代分析自适应最优控制的稳定性，考虑了近似误差的影响，提供了吸引域的估计，确保初始条件在该域内时轨迹保持有效。

随机化最小二乘值迭代的频率学派遗憾界

Frequentist Regret Bounds for Randomized Least-Squares Value Iteration

针对有限时域强化学习的探索-利用困境，本文提出了一种乐观初始化的 RLSVI 变体，并在低秩转移动态假设下首次证明了带函数逼近的随机化探索的频率学派遗憾上界。

Robust Value Iteration for Continuous Control Tasks

When transferring a control policy from simulation to a physical system, the policy needs to be robust to variations in the dynamics to perform well. Commonly, the optimal policy overfits to the approximate model and the corresponding state-distribution, often resulting in failure to trasnfer underlying distributional shifts. In this paper, we present Robust Fitted Value Iteration, which uses dynamic programming to compute the optimal value function on the compact state domain and incorporates adversarial perturbations of the system dynamics. The adversarial perturbations encourage a optimal p…

面向离线强化学习的悲观非线性最小二乘值迭代

Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning

针对非线性函数逼近的离线强化学习缺乏实例依赖悔界保证的问题，提出预言机高效的PNLSVI算法，含三个创新组件，实现紧密的函数类复杂度依赖悔界，将相关结果扩展到更通用框架。

用于学习可并发执行的机器人控制任务的值迭代

Value Iteration for Learning Concurrently Executable Robotic Control Tasks

提出基于强化学习的值迭代方法，通过定义任务独立性并优化成本泛函，训练冗余机器人并发执行多任务且最小化高优先级任务干扰。