Foresee then Evaluate: Decomposing Value Estimation with Latent Future Prediction
Tianjin University（天津大学）；Huawei（华为）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；The Chinese University of Hong Kong（香港中文大学）
Value function is the central notion of Reinforcement Learning (RL). Value estimation, especially with function approximation, can be challenging since it involves the stochasticity of environmental dynamics and reward signals that can be sparse and delayed in some cases. A typical model-free RL algorithm usually estimates the values of a policy by Temporal Difference (TD) or Monte Carlo (MC) algorithms directly from rewards, without explicitly taking dynamics into consideration. In this paper, we propose Value Decomposition with Future Prediction (VDFP), providing an explicit two-step understanding of the value estimation process: 1) first foresee the latent future, 2) and then evaluate it. We analytically decompose the value function into a latent future dynamics part and a policy-independent trajectory return part, inducing a way to model latent dynamics and returns separately in value estimation. Further, we derive a practical deep RL algorithm, consisting of a convolutional model to learn compact trajectory representation from past experiences, a conditional variational auto-encoder to predict the latent future dynamics and a convex return model that evaluates trajectory representation. In experiments, we empirically demonstrate the effectiveness of our approach for both off-policy and on-policy RL in several OpenAI Gym continuous control tasks as well as a few challenging variants with delayed reward.
Inferring Latent Temporal Sparse Coordination Graph for Multi-Agent Reinforcement Learning
Australian Artificial Intelligence Institute (AAII), Faculty of Engineering and Information Technology, University of Technology Sydney（澳大利亚人工智能研究所（AAII），悉尼科技大学工程与信息技术学院）
Decomposing feature-level variation with Covariate Gaussian Process Latent Variable Models
University of Oxford（牛津大学）；University of British Columbia（不列颠哥伦比亚大学）；BC Cancer Agency（不列颠哥伦比亚癌症机构）；UBC Data Science Institute（不列颠哥伦比亚大学数据科学研究所）；The Alan Turing Institute（艾伦·图灵研究所）；University of Birmingham（伯明翰大学）
Towards Governing Agent's Efficacy: Action-Conditional $β$-VAE for Deep Transparent Reinforcement Learning
Seoul National University（首尔国立大学）；Samsung Electronics（三星电子）
我们在强化学习（RL）的设置中解决了深度神经网络的黑盒问题，其中神经代理以不可控的方式学习最大化奖励收益。当交互环境包括广阔的状态空间时，这种学习方法是有风险的，因为几乎不可能预见所有不想要的结果并预先用负奖励惩罚它们。与以前的作品中学习的神经功能的反向分析不同，我们提出的方法\nj{tackles the blackbox issue by encouraging}是一个RL策略网络，通过实现一个解纠缠的表示学习方法来学习可解释的潜在特征。为此，我们的方法允许RL代理通过将其影响与不可控的环境因素区分开来来理解自我效能，这与人类理解场景的方式非常相似。我们的实验结果表明，学习的潜在因素不仅是可解释的，而且还可以建模整个访问状态空间的分布与特定的动作条件。我们已经实验表明，拟议结构的这一特征可以导致对RL代理的期望行为进行事后治理。
Semantically Decomposing the Latent Spaces of Generative Adversarial Networks
University of California, San Diego（加利福尼亚大学圣地亚哥分校）；Carnegie Mellon University（卡内基梅隆大学）；Amazon AI（亚马逊人工智能）；Stanford University（斯坦福大学）；University of California San Diego（加利福尼亚大学圣地亚哥分校）
Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea（首尔国立大学电气与计算机工程系）；Department of Automotive Engineering, Ajou University, Gyeonggi-do, South Korea（全州大学汽车工程系）
Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus
The Hong Kong University of Science and Technology（香港科学与技术大学）；The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））
Understanding the Staged Dynamics of Transformers in Learning Latent Structure
Department of Computing Science, University of Alberta, Edmonton, Canada（计算科学系，阿尔伯塔大学，爱德蒙顿，加拿大）；Department of Psychology, University of Alberta, Edmonton, Canada（心理学系，阿尔伯塔大学，爱德蒙顿，加拿大）
