Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等技术学院）；Instituto de Telecomunicações（电信研究所）；IDMEC, Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等技术学院 IDMEC）
João Pedro Araújo, Mário A. T. Figueiredo, Miguel Ayala Botto
备注：
29 pages, 13 figures. arXiv admin note: substantial text overlap with arXiv:2007.06741
This paper evaluates adaptive Q-learning (AQL) and single-partition adaptive Q-learning (SPAQL), two algorithms for efficient model-free episodic reinforcement learning (RL), in two classical control problems (Pendulum and Cartpole). AQL adaptively partitions the state-action space of a Markov decision process (MDP), while learning the control policy, i. e., the mapping from states to actions. The main difference between AQL and SPAQL is that the latter learns time-invariant policies, where the mapping from states to actions does not depend explicitly on the time step. This paper also proposes the SPAQL with terminal state (SPAQL-TS), an improved version of SPAQL tailored for the design of regulators for control problems. The time-invariant policies are shown to result in a better performance than the time-variant ones in both problems studied. These algorithms are particularly fitted to RL problems where the action space is finite, as is the case with the Cartpole problem. SPAQL-TS solves the OpenAI Gym Cartpole problem, while also displaying a higher sample efficiency than trust region policy optimization (TRPO), a standard RL algorithm for solving control tasks. Moreover, the policies learned by SPAQL are interpretable, while TRPO policies are typically encoded as neural networks, and therefore hard to interpret. Yielding interpretable policies while being sample-efficient are the major advantages of SPAQL.
Adaptive Regret for Control of Time-Varying Dynamics
UC Berkeley（加州大学伯克利分校）；Google AI Princeton（谷歌人工智能普林斯顿）；Princeton University（普林斯顿大学）
本文研究具有时变线性动态系统的在线控制问题。这是一个一般的配方，是由使用局部线性化控制的非线性动力系统的动机。为了在不断变化的环境中陈述有意义的保证，我们将{ \it自适应后悔}的度量引入到控制领域。该度量最初在在线学习中研究，在{ \it any interval in time}上根据事后对最佳策略的后悔来测量性能，从而捕获控制器对变化的动态的适应。我们的主要贡献是一种新的有效的元算法：它将一个控制器与次线性遗憾界到一个与次线性{ \it自适应遗憾}界的设置时变线性动力系统。主要的技术创新是第一个自适应遗憾界的更一般的框架，在线凸优化与记忆。此外，我们给出了一个下界，表明我们获得的自适应遗憾界是接近紧的一般框架。
Adaptive Control of SE(3) Hamiltonian Dynamics with Learned Disturbance Features
University of California San Diego（加州大学圣地亚哥分校）
自适应控制是机器人在快速变化的操作条件下可靠自主的关键组成部分。自适应控制设计受益于干扰模型，这在实践中往往是不可用的。这促使使用机器学习技术来离线地从训练数据学习干扰特征，其随后可以用于在线地补偿干扰。本文开发了几何自适应控制与学习的干扰模型的刚体系统，如地面，空中和水下航行器，满足汉密尔顿的运动方程的$SE(3)$流形。我们的设计包括一个\emph{offline disturbance model identification stage}，使用汉密尔顿为基础的神经常微分方程（ODE）网络训练的状态控制轨迹数据，和一个\emph{online adaptive control stage}，估计和补偿的基础上的几何跟踪误差的干扰。我们证明了我们的自适应几何控制器的轨迹跟踪仿真全驱动摆和欠驱动四旋翼系统。
