We study the link between generalization and interference in temporal-difference (TD) learning. Interference is defined as the inner product of two different gradients, representing their alignment. This quantity emerges as being of interest from a variety of observations about neural networks, parameter sharing and the dynamics of learning. We find that TD easily leads to low-interference, under-generalizing parameters, while the effect seems reversed in supervised learning. We hypothesize that the cause can be traced back to the interplay between the dynamics of interference and bootstrapping. This is supported empirically by several observations: the negative relationship between the generalization gap and interference in TD, the negative effect of bootstrapping on interference and the local coherence of targets, and the contrast between the propagation rate of information in TD(0) versus TD($λ$) and regression tasks such as Monte-Carlo policy evaluation. We hope that these new findings can guide the future discovery of better bootstrapping methods.
Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferences
Shanghai University of Finance and Economics（上海财经大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Bytedance（字节跳动）；London School of Economics and Political Science（伦敦政治经济学院）
Temporal Latent Variable Structural Causal Model for Causal Discovery under External Interferences
School of Computer Science, Guangdong University of Technology（广东工业大学计算机科学学院）；Peng Cheng Laboratory（鹏城实验室）；Machine Learning Department, Mohamed bin Zayed University of Artificial Intelligence（Mohamed bin Zayed人工智能大学机器学习系）；College of Mathematics and Computer Science, Shantou University（汕头大学数学与计算机科学学院）
Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learning
McGill University（麦吉尔大学）；Microsoft Research Montreal（微软研究院蒙特利尔分部）；Université de Montréal（蒙特利尔大学）；Mila（米拉研究所）；Sony AI（索尼人工智能公司）；Google DeepMind（谷歌DeepMind）
STRAP: Spatio-Temporal Pattern Retrieval for Out-of-Distribution Generalization
City University of Hong Kong（香港城市大学）；City University of Hong Kong (Dongguan)（香港城市大学（东莞））；Northeastern University（东北大学）；The Hong Kong Polytechnic University（香港理工大学）；University of Electronic Science and Technology of China（电子科技大学）；SLAI
