Model-based reinforcement learning uses models to plan, where the predictions and policies of an agent can be improved by using more computation without additional data from the environment, thereby improving sample efficiency. However, learning accurate estimates of the model is hard. Subsequently, the natural question is whether we can get similar benefits as planning with model-free methods. Experience replay is an essential component of many model-free algorithms enabling sample-efficient learning and stability by providing a mechanism to store past experiences for further reuse in the gradient computational process. Prior works have established connections between models and experience replay by planning with the latter. This involves increasing the number of times a mini-batch is sampled and used for updates at each step (amount of replay per step). We attempt to exploit this connection by doing a systematic study on the effect of varying amounts of replay per step in a well-known model-free algorithm: Deep Q-Network (DQN) in the Mountain Car environment. We empirically show that increasing replay improves DQN's sample efficiency, reduces the variation in its performance, and makes it more robust to change in hyperparameters. Altogether, this takes a step toward a better algorithm for deployment.
Augmented Replay Memory in Reinforcement Learning With Continuous Control
Artificial Intelligence Center(人工智能中心);Faculty of Electrical Engineering(电气工程学院);Czech Technical University in Prague(捷克理工大学);Dipartimento di Elettronica, Informazione e Bioingegneria(电子、信息与生物工程系);Politecnico di Milano(米兰理工大学)
REPLAY: Modeling Time-Varying Temporal Regularities of Human Mobility for Location Prediction over Sparse Trajectories
State Key Laboratory of Internet of Things for Smart City and Department of Computer and Information Science, University of Macau(物联网智能城市国家重点实验室和计算机与信息科学系,澳门大学);BNU-HKBU United International College(北京师范大学-香港浸会大学联合国际学院);Bern University of Applied Sciences(伯尔尼应用科学大学);University of Fribourg(弗里堡大学)
Learning Design and Construction with Varying-Sized Materials via Prioritized Memory Resets
Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);ByteDance AI Lab(字节跳动人工智能实验室);University of California Santa Barbara(加州大学圣巴巴拉分校);Shanghai Qi Zhi Institute(上海期智研究院)
Provable Effects of Data Replay in Continual Learning: A Feature Learning Perspective
School of Information Science and Technology, USTC and Institute of Artificial Intelligence, HCNSC(信息科学与技术学院,中国科学技术大学及人工智能研究所);Department of Computer Science and Engineering, SUNY at Buffalo(计算机科学与工程系,纽约州立大学布法罗分校)
FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning
The Hong Kong Polytechnic University(香港理工大学);ShineMo Ltd., China(中国 ShineMo 公司);Solar System of OVB, Tencent, China(腾讯 OVB 太阳系中国);Peking University(北京大学);University College London(伦敦大学学院);University of Illinois Chicago(伊利诺伊大学芝加哥分校)
A New Algorithm for Non-stationary Contextual Bandits: Efficient, Optimal, and Parameter-free
University of Southern California(南加州大学)
我们提出了第一个上下文强盗算法,是无参数,高效,最佳的动态遗憾。具体来说,我们的算法实现了动态遗憾$\mathcal{O}(\min\{\sqrt{ST}, Δ^{\frac{1}{3}}T^{\frac{2}{3}}\})$的上下文强盗问题与$T$轮,$S$开关和$Δ$总变化的数据分布。重要的是,我们的算法是自适应的,不需要提前知道$S$或$Δ$,并可以有效地实现假设访问一个ERM oracle。我们的结果严格地改进了(Luo et al.,2018),并大大推广和改进了(Auer等人,2018)的$\mathcal{O}(\sqrt{ST})$结果,该结果仅适用于没有上下文信息的双臂强盗问题。我们的算法的关键新颖性是引入重放阶段,在该阶段中,算法根据其先前的决定在一定的时间内采取行动,以检测非平稳性,同时保持探索和开发之间的良好平衡。
FETCH: A Memory-Efficient Replay Approach for Continual Learning in Image Classification
Faculty of Electrical Engineering and Information Technology, Chemnitz University of Technology(德国 Chemnitz 工业大学 电气工程与信息技术学院);Institute for Computational Visualistics, University of Koblenz(科布伦茨大学 计算视觉学研究所)
Catastrophic Forgetting Mitigation via Discrepancy-Weighted Experience Replay
Institute of Software Chinese Academy of Science, Beijing, China(中国科学院软件研究所);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学);Advanced Institute of Big Data, Beijing, China(大数据先进研究院)