Task Aware Dreamer for Task Generalization in Reinforcement Learning
Department of Computer Science and Technology（计算机科学与技术系）；Beijing National Research Center for Information Science and Technology（北京信息科学国家研究中心）；Tsinghua-Bosch Joint Center for Machine Learning, Institute for Artificial Intelligence（清华大学-博世联合机器学习中心，人工智能研究院）；Tsinghua University（清华大学）
Chengyang Ying, Xinning Zhou, Zhongkai Hao, Hang Su, Songming Liu, Dong Yan, Jun Zhu
A long-standing goal of reinforcement learning is to acquire agents that can learn on training tasks and generalize well on unseen tasks that may share a similar dynamic but with different reward functions. The ability to generalize across tasks is important as it determines an agent's adaptability to real-world scenarios where reward mechanisms might vary. In this work, we first show that training a general world model can utilize similar structures in these tasks and help train more generalizable agents. Extending world models into the task generalization setting, we introduce a novel method named Task Aware Dreamer (TAD), which integrates reward-informed features to identify consistent latent characteristics across tasks. Within TAD, we compute the variational lower bound of sample data log-likelihood, which introduces a new term designed to differentiate tasks using their states, as the optimization objective of our reward-informed world models. To demonstrate the advantages of the reward-informed policy in TAD, we introduce a new metric called Task Distribution Relevance (TDR) which quantitatively measures the relevance of different tasks. For tasks exhibiting a high TDR, i.e., the tasks differ significantly, we illustrate that Markovian policies struggle to distinguish them, thus it is necessary to utilize reward-informed policies in TAD. Extensive experiments in both image-based and state-based tasks show that TAD can significantly improve the performance of handling different tasks simultaneously, especially for those with high TDR, and display a strong generalization ability to unseen tasks.
Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning
Graduate School of Informatics, Kyoto University, Kyoto, Japan（京都大学信息学研究生院）；Research Organization of Science and Technology, Ritsumeikan University, Shiga, Japan（立命馆大学科学技术研究机构）
DreamerAD: Efficient Reinforcement Learning via Latent World Model for Autonomous Driving
Institute of Automation, CAS（中国科学院自动化研究所）；Chongqing Chang’an Technology Co., Ltd（重庆长安科技有限公司）；School of Advanced Interdisciplinary Sciences, UCAS（中国科学院大学先进交叉科学学院）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）
The Effectiveness of World Models for Continual Reinforcement Learning
University of Oxford（牛津大学）；Warsaw University of Technology（华沙理工大学）；Jagiellonian University（雅盖隆大学）；Polish Academy of Sciences（波兰科学院）；Institute of Theoretical and Applied Informatics, PAS（波兰科学院理论与应用信息研究所）
Mind Dreamer: Untethering Imagination via Active Causal Intervention on Latent Manifolds
Center for Brain-Inspired Computing Research, Department of Precision Instrument, Tsinghua University, Beijing, China（脑启发计算研究中心，精密仪器系，清华大学，北京，中国）；College of Computer Science and Technology, Zhejiang University, Hangzhou, China（计算机科学与技术学院，浙江大学，杭州，中国）；Pen-Tung Sah Institute of Micro-Nano Science and Technology, Xiamen University, Xiamen, China（彭途萨微纳米科学与技术研究院，厦门大学，厦门，中国）
Dynamic Update-to-Data Ratio: Minimizing World Model Overfitting
University of Freiburg（弗莱堡大学）；University of Technology Nuremberg（纽伦堡工业大学）
基于验证集性能的早期停止是在监督学习的上下文中找到欠拟合和过拟合之间的正确平衡的流行方法。然而，在强化学习中，即使对于诸如世界模型学习的监督子问题，早期停止也不适用，因为数据集是不断发展的。作为一种解决方案，我们提出了一种新的通用方法，该方法在训练过程中动态调整更新数据（UTD）比率，该方法基于对未用于训练的连续收集的经验的一个小子集的欠拟合和过拟合检测。我们将我们的方法应用于DreamerV2，这是一种最先进的基于模型的强化学习算法，并在DeepMind Control Suite和Atari $100$ k基准测试中对其进行了评估。结果表明，可以更好地平衡下和高估，通过调整UTD比与我们的方法相比，在DreamerV2的默认设置，它是一个广泛的超参数搜索，这是不可行的许多应用程序的竞争力。我们的方法消除了手动设置UTD超参数的需要，甚至导致了对其他学习相关超参数的更高鲁棒性，进一步减少了必要的调整量。
PIGDreamer: Privileged Information Guided World Models for Safe Partially Observable Reinforcement Learning
School of Computer Science, University of Beihang, Beijing, China（北京航空航天大学计算机学院）；School of Computer Science, Chinese University of Hong Kong, Hongkong, China（香港中文大学计算机学院）；JD Explore Academy, Beijing, China（京东探索研究院）；North Automatic Control Institute, Taiyuan, China（太原北自动控制研究所）
Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methods
Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；University of Alberta（阿尔伯塔大学）；Universite de Montreal（蒙特利尔大学）；Microsoft（微软）；Ecole Polytechnique de Montreal（蒙特利尔高等理工学院）
GIRL: Generative Imagination Reinforcement Learning via Information-Theoretic Hallucination Control
Department of Computer Science and Engineering, Visvesvaraya Technological University (VTU), Belagavi, India（维斯瓦拉亚科技大学计算机科学与工程系，贝拉加维，印度）；Aliens on Earth (AoE) Autonomous Research Group, Belagavi, India（地球外星人自主研究组，贝拉加维，印度）
