A Monte Carlo AIXI Approximation

Reinforcement Learning via AIXI Approximation

This paper introduces a principled approach for the design of a scalable general reinforcement learning agent. This approach is based on a direct approximation of AIXI, a Bayesian optimality notion for general reinforcement learning agents. Previously, it has been unclear whether the theory of AIXI could motivate the design of practical algorithms. We answer this hitherto open question in the affirmative, by providing the first computationally feasible approximation to the AIXI agent. To develop our approximation, we introduce a Monte Carlo Tree Search algorithm along with an agent-specific ex…

A Monte Carlo Algorithm for Universally Optimal Bayesian Sequence Prediction and Planning

The aim of this work is to address the question of whether we can in principle design rational decision-making agents or artificial intelligences embedded in computable physics such that their decisions are optimal in reasonable mathematical senses. Recent developments in rare event probability estimation, recursive bayesian inference, neural networks, and probabilistic planning are sufficient to explicitly approximate reinforcement learners of the AIXI style with non-trivial model classes (here, the class of resource-bounded Turing machines). Consideration of the effects of resource limitatio…

Generalised Discount Functions applied to a Monte-Carlo AImu Implementation

In recent years, work has been done to develop the theory of General Reinforcement Learning (GRL). However, there are few examples demonstrating these results in a concrete way. In particular, there are no examples demonstrating the known results regarding gener- alised discounting. We have added to the GRL simulation platform AIXIjs the functionality to assign an agent arbitrary discount functions, and an environment which can be used to determine the effect of discounting on an agent's policy. Using this, we investigate how geometric, hyperbolic and power discounting affect an informed agent…

蒙特卡洛搜索算法发现蒙特卡洛树搜索探索项

Monte Carlo Search Algorithms Discovering Monte Carlo Tree Search Exploration Terms

本研究提出用蒙特卡洛搜索自动设计PUCT、SHUSS两种蒙特卡洛树搜索算法的根探索项，在32次评估的低搜索预算下，使二者性能比肩常规PUCT，为组合问题搜索算法优化提供新路径。

蒙特卡洛树搜索只是全访问蒙特卡洛控制吗？

Is Monte Carlo Tree Search Just Every-Visit Monte Carlo Control?

本文论证蒙特卡洛树搜索（MCTS）与全访问蒙特卡洛控制本质等价，仅表述术语不同，MCTS可简化为轨迹采样与全访问蒙特卡洛更新两个操作，旨在明确二者的等价关系。

Policy Learning and Evaluation with Randomized Quasi-Monte Carlo

Reinforcement learning constantly deals with hard integrals, for example when computing expectations in policy evaluation and policy iteration. These integrals are rarely analytically solvable and typically estimated with the Monte Carlo method, which induces high variance in policy values and gradients. In this work, we propose to replace Monte Carlo samples with low-discrepancy point sets. We combine policy gradient methods with Randomized Quasi-Monte Carlo, yielding variance-reduced formulations of policy gradient and actor-critic algorithms. These formulations are effective for policy eval…