Equilibrium Propagation (EP) is a powerful and more bio-plausible alternative to conventional learning frameworks such as backpropagation. The effectiveness of EP stems from the fact that it relies only on local computations and requires solely one kind of computational unit during both of its training phases, thereby enabling greater applicability in domains such as bio-inspired neuromorphic computing. The dynamics of the model in EP is governed by an energy function and the internal states of the model consequently converge to a steady state following the state transition rules defined by the same. However, by definition, EP requires the input to the model (a convergent RNN) to be static in both the phases of training. Thus it is not possible to design a model for sequence classification using EP with an LSTM or GRU like architecture. In this paper, we leverage recent developments in modern hopfield networks to further understand energy based models and develop solutions for complex sequence classification tasks using EP while satisfying its convergence criteria and maintaining its theoretical similarities with recurrent backpropagation. We explore the possibility of integrating modern hopfield networks as an attention mechanism with convergent RNN models used in EP, thereby extending its applicability for the first time on two different sequence classification tasks in natural language processing viz. sentiment analysis (IMDB dataset) and natural language inference (SNLI dataset).
Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learning
MRC Brain Networks Dynamics Unit(英国医学研究理事会脑网络动力学部门);University of Oxford(牛津大学)
Lagrangian-based Equilibrium Propagation: generalisation to arbitrary boundary conditions & equivalence with Hamiltonian Echo Learning
University of Groningen(格罗宁根大学);Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStAL(里尔大学、法国国家信息与自动化研究所、法国国家科学研究中心、中央理工-里尔高等电力学院,UMR 9189 - CRIStAL);Centrum Wiskunde & Informatica(荷兰国家数学与计算机科学研究中心);The University of Sheffield(谢菲尔德大学)
Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias
Université Paris-Saclay(巴黎萨克雷大学);CNRS(法国国家科学研究中心);Unité Mixte de Physique(联合物理实验室);Thales(泰雷兹集团);Mila(米拉研究所);Université de Montréal(蒙特利尔大学);Canadian Institute for Advanced Research(加拿大高等研究院);Google(谷歌)
Equilibrium Propagation with Continual Weight Updates
Centre de Nanosciences et de Nanotechnologies, Université Paris-Saclay(巴黎萨克雷大学纳米科学与纳米技术中心);Unité Mixte de Physique, CNRS, Thales, Université Paris-Saclay(巴黎萨克雷大学、法国国家科学研究中心、泰雷兹集团联合物理单位);Mila, Université de Montréal(蒙特利尔大学米拉研究所);Canadian Institute for Advanced Research(加拿大高等研究院)
Continual Weight Updates and Convolutional Architectures for Equilibrium Propagation
Centre de Nanosciences et de Nanotechnologies, Université Paris-Saclay(巴黎萨克雷大学纳米科学与纳米技术中心);Unité Mixte de Physique, CNRS, Thales, Université Paris-Saclay(巴黎萨克雷大学、法国国家科学研究中心、泰雷兹集团联合物理单位);Canadian Institute for Advanced Research(加拿大高等研究院);Mila, Université de Montréal(蒙特利尔大学米拉研究所)