Doubly robust methods hold considerable promise for off-policy evaluation in Markov decision processes (MDPs) under sequential ignorability: They have been shown to converge as $1/\sqrt{T}$ with the horizon $T$, to be statistically efficient in large samples, and to allow for modular implementation where preliminary estimation tasks can be executed using standard reinforcement learning techniques. Existing results, however, make heavy use of a strong distributional overlap assumption whereby the stationary distributions of the target policy and the data-collection policy are within a bounded factor of each other -- and this assumption is typically only credible when the state space of the MDP is bounded. In this paper, we re-visit the task of off-policy evaluation in MDPs under a weaker notion of distributional overlap, and introduce a class of truncated doubly robust (TDR) estimators which we find to perform well in this setting. When the distribution ratio of the target and data-collection policies is square-integrable (but not necessarily bounded), our approach recovers the large-sample behavior previously established under strong distributional overlap. When this ratio is not square-integrable, TDR is still consistent but with a slower-than-$1/\sqrt{T}$-rate; furthermore, this rate of convergence is minimax over a class of MDPs defined only using mixing conditions. We validate our approach numerically and find that, in our experiments, appropriate truncation plays a major role in enabling accurate off-policy evaluation when strong distributional overlap does not hold.
Statistical Efficiency of Distributional Temporal Difference Learning and Freedman's Inequality in Hilbert Spaces
School of Mathematical Sciences, Peking University（北京大学数学科学学院）；School of Statistics and Data Science, Shanghai University of Finance and Economics（上海财经大学统计与数据科学学院）
A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximation
School of Mathematical Sciences, Peking University（北京大学数学科学学院）；School of Statistics and Data Science, Shanghai University of Finance and Economics（上海财经大学统计与数据科学学院）
Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectories
City University of Hong Kong（香港城市大学）；Center for Advanced Intelligence Project, RIKEN（RIKEN高级智能研究中心）；The Institute of Statistical Mathematics（统计数学研究所）；University of Sydney（悉尼大学）；CFAR and IHPC, Agency for Science, Technology and Research (A*STAR)（A*STAR的CFAR和IHPC）；Nanyang Technological University（南洋理工大学）；The University of Tokyo（东京大学）
On the Foundation of Distributionally Robust Reinforcement Learning
Department of Industrial and Systems Engineering, University of Southern California（工业与系统工程系，南加州大学）；Industrial Engineering and Decision Analytics, Hong Kong University of Science and Technology（工业工程与决策分析，香港科技大学）；Management Science and Engineering, Stanford University（管理科学与工程，斯坦福大学）；Stern School of Business, New York University（斯特恩商学院，纽约大学）
Distributionally Robust Optimization (DRO) has enabled to prove the equivalence between robustness and regularization in classification and regression, thus providing an analytical reason why regularization generalizes well in statistical learning. Although DRO's extension to sequential decision-making overcomes $\textit{external uncertainty}$ through the robust Markov Decision Process (MDP) setting, the resulting formulation is hard to solve, especially on large domains. On the other hand, existing regularization methods in reinforcement learning only address $\textit{internal uncertainty}$ d…
Department of Mathematics, Linnæus University（林奈大学数学系）
In this work we continue to build upon recent advances in reinforcement learning for finite Markov processes. A common approach among previous existing algorithms, both single-actor and distributed, is to either clip rewards or to apply a transformation method on Q-functions to handle a large variety of magnitudes in real discounted returns. We theoretically show that one of the most successful methods may not yield an optimal policy if we have a non-deterministic process. As a solution, we argue that distributional reinforcement learning lends itself to remedy this situation completely. By th…
In dynamic programming (DP) and reinforcement learning (RL), an agent learns to act optimally in terms of expected long-term return by sequentially interacting with its environment modeled by a Markov decision process (MDP). More generally in distributional reinforcement learning (DRL), the focus is on the whole distribution of the return, not just its expectation. Although DRL-based methods produced state-of-the-art performance in RL with function approximation, they involve additional quantities (compared to the non-distributional setting) that are still not well understood. As a first contr…
Singapore Management University（新加坡管理大学）；Massachusetts Institute of Technology（麻省理工学院）
We study the relation between different Markov Decision Process (MDP) frameworks in the machine learning and econometrics literatures, including the standard MDP, the entropy and general regularized MDP, and stochastic MDP, where the latter is based on the assumption that the reward function is stochastic and follows a given distribution. We show that the entropy-regularized MDP is equivalent to a stochastic MDP model, and is strictly subsumed by the general regularized MDP. Moreover, we propose a distributional stochastic MDP framework by assuming that the distribution of the reward function …
We propose a distributionally robust return-risk model for Markov decision processes (MDPs) under risk and reward ambiguity. The proposed model optimizes the weighted average of mean and percentile performances, and it covers the distributionally robust MDPs and the distributionally robust chance-constrained MDPs (both under reward ambiguity) as special cases. By considering that the unknown reward distribution lies in a Wasserstein ambiguity set, we derive the tractable reformulation for our model. In particular, we show that that the return-risk model can also account for risk from uncertain…
We introduce a distributional method for learning the optimal policy in risk averse Markov decision process with finite state action spaces, latent costs, and stationary dynamics. We assume sequential observations of states, actions, and costs and assess the performance of a policy using dynamic risk measures constructed from nested Kusuoka-type conditional risk mappings. For such performance criteria, randomized policies may outperform deterministic policies, therefore, the candidate policies lie in the d-dimensional simplex where d is the cardinality of the action space. Existing risk averse…
Topology-Aware State Abstraction with Tangle Cores for Markov Decision Processes
Department of Computer Science, Iowa State University（计算机科学系，爱荷华州立大学）；Department of Civil, Construction & Environmental Engineering, Iowa State University（土木、建设与环境工程系，爱荷华州立大学）
