Efficient Optimistic Exploration in Linear-Quadratic Regulators via Lagrangian Relaxation

A Unifying View of Optimism in Episodic Reinforcement Learning

The principle of optimism in the face of uncertainty underpins many theoretically successful reinforcement learning algorithms. In this paper we provide a general framework for designing, analyzing and implementing such algorithms in the episodic reinforcement learning problem. This framework is built upon Lagrangian duality, and demonstrates that every model-optimistic algorithm that constructs an optimistic MDP has an equivalent representation as a value-optimistic dynamic programming algorithm. Typically, it was thought that these two classes of algorithms were distinct, with model-optimist…

乐观在线LQR via 内在奖励

Optimistic Online LQR via Intrinsic Rewards

本文提出IR-LQR算法，通过内在奖励和方差正则化提升不确定性驱动探索，实现最优最坏情况后悔率√T，实验验证其在飞行器pitch角控制中的有效性。

Verification and Parameter Synthesis for Stochastic Systems using Optimistic Optimization

We present an algorithm for formal verification and parameter synthesis of continuous state-space Markov chains. This class of problems captures the design and analysis of a wide variety of autonomous and cyber-physical systems defined by nonlinear and black-box modules. In order to solve these problems, one has to maximize certain probabilistic objective functions overall choices of initial states and parameters. In this paper, we identify the assumptions that make it possible to view this problem as a multi-armed bandit problem. Based on this fresh perspective, we propose an algorithm (HOO-M…

通过拉格朗日方法学习探索：在未知线性约束下的多臂老虎机

Learning to Explore with Lagrangians for Bandits under Unknown Linear Constraints

本文提出LATS和LAGEX两种方法，通过拉格朗日松弛优化多臂老虎机在未知线性约束下的探索效率，实现渐近最优样本复杂度。

Exploration in Model-based Reinforcement Learning with Randomized Reward

Model-based Reinforcement Learning (MBRL) has been widely adapted due to its sample efficiency. However, existing worst-case regret analysis typically requires optimistic planning, which is not realistic in general. In contrast, motivated by the theory, empirical study utilizes ensemble of models, which achieve state-of-the-art performance on various testing environments. Such deviation between theory and empirical study leads us to question whether randomized model ensemble guarantee optimism, and hence the optimal worst-case regret? This paper partially answers such question from the perspec…

确定性动力学下基于线性Bellman完整性的计算高效强化学习

Computationally Efficient RL under Linear Bellman Completeness for Deterministic Dynamics

针对线性Bellman完整设置，提出一种基于随机噪声注入和零空间设计的计算高效强化学习算法，适用于大动作空间和确定性动力学，解决了误差放大问题。