Model-Free Adaptive Optimal Control of Episodic Fixed-Horizon Manufacturing Processes using Reinforcement Learning
Institute Intelligent Systems Research Group, Karlsruhe University of Applied Sciences（智能系统研究组，卡尔斯鲁厄应用科学大学）；Institute for Applied Materials (IAM-CMS), Karlsruhe Institute of Technology（应用材料研究所（IAM-CMS），卡尔斯鲁厄理工学院）
Johannes Dornheim, Norbert Link, Peter Gumbsch
备注：
Journal preprint version
发表信息：
International Journal of Control, Automation and Systems (2019)
A self-learning optimal control algorithm for episodic fixed-horizon manufacturing processes with time-discrete control actions is proposed and evaluated on a simulated deep drawing process. The control model is built during consecutive process executions under optimal control via reinforcement learning, using the measured product quality as reward after each process execution. Prior model formulation, which is required by state-of-the-art algorithms from model predictive control and approximate dynamic programming, is therefore obsolete. This avoids several difficulties namely in system identification, accurate modelling, and runtime complexity, that arise when dealing with processes subject to nonlinear dynamics and stochastic influences. Instead of using pre-created process and observation models, value function-based reinforcement learning algorithms build functions of expected future reward, which are used to derive optimal process control decisions. The expectation functions are learned online, by interacting with the process. The proposed algorithm takes stochastic variations of the process conditions into account and is able to cope with partial observability. A Q-learning-based method for adaptive optimal control of partially observable episodic fixed-horizon manufacturing processes is developed and studied. The resulting algorithm is instantiated and evaluated by applying it to a simulated stochastic optimal control problem in metal sheet deep drawing.
Is Long Horizon Reinforcement Learning More Difficult Than Short Horizon Reinforcement Learning?
Carnegie Mellon University（卡内基梅隆大学）；Institute for Advanced Study（高等研究院）；University of California, Los Angles（加州大学洛杉矶分校）；University of Washington, Seattle（华盛顿大学西雅图分校）；Microsoft Research（微软研究院）
On the Convergence of Monte Carlo UCB for Random-Length Episodic MDPs
SFSC of AI and DL, NYU Shanghai（纽约大学上海人工智能与深度学习科学与技术中心）；New York University（纽约大学）；New York University Shanghai（纽约大学上海分校）；New York University Abu Dhabi（纽约大学阿布扎比分校）
Why is Posterior Sampling Better than Optimism for Reinforcement Learning?
Stanford University（斯坦福大学）；Deepmind
计算结果表明，后验采样强化学习（PSRL）显着优于乐观驱动的算法，如UCRL 2。我们提供了深入了解这种性能提升的程度和驱动它的现象。我们利用这种洞察力，建立一个$\tilde{O}(H\sqrt{SAT})$贝叶斯期望遗憾界PSRL在有限地平线情节马尔可夫决策过程中，其中$H$是地平线，$S$是状态的数量，$A$是行动的数量和$T$是经过的时间。这改进了任何强化学习算法的$\tilde{O}(H S \sqrt{AT})$的最佳前界。
Learning Long-Term Reward Redistribution via Randomized Return Decomposition
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Shanghai Jiao Tong University（上海交通大学）；BIMSA（北京雁栖湖应用数学研究院）；AIR, Tsinghua University（清华大学智能产业研究院）；HeliXon Limited（华深智药）
Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Bounds
Institute for Computational and Mathematical Engineering, Stanford University（斯坦福大学计算与数学工程研究所）；Department of Computer Science, Stanford University（斯坦福大学计算机科学系）
Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learning
UC Santa Barbara（加州大学圣巴巴拉分校）
我们考虑了强化学习的离线策略评估问题，其目标是使用通过运行日志策略$μ$收集的离线数据来估计目标策略$π$的预期奖励。标准的基于重要性抽样的方法在这个问题上受到方差的影响，该方差随时间范围$H$呈指数级增长，这激发了人们对打破“地平线诅咒”的替代方案的兴趣（Liu et al. 2018，Xie et al. 2019）。特别是，它表明，边缘化的重要性采样（MIS）的方法可以用来实现估计误差的顺序$O(H^3/ n)$的均方误差（MSE）下的情节马尔可夫决策过程模型与有限的状态和潜在的无限的行动。然而，MSE界仍然是$H$的因子，远离$Ω(H^2/n)$阶的Cramer-Rao下界。在本文中，我们证明了一个简单的修改MIS估计，我们可以渐近地达到Cramer-Rao下界，只要作用空间是有限的。我们还提供了一个一般的方法来构建MIS估计与高概率误差界。
High-Probability Nash Regret for Decentralized Learning in Markov $α$-Potential Games: Episodic and Fully Online Asynchronous Algorithms with Applications to Markov Congestion Games
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Hierarchical Decentralized Stochastic Control for Cyber-Physical Systems
Department of Electrical Engineering and Computer Science, University of Ottawa（电气与计算机科学系，渥太华大学）；Department of Electrical Engineering, Eindhoven University of Technology (TU/e)（电气工程系，埃因霍温理工大学）；Department of Electrical Engineering, Indian Institute of Technology Madras（电气工程系，印度理工学院马德拉斯学院）
