arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.09626cs.NEcs.AIcs.LG
收藏

基于平衡传播的序列学习

Sequence Learning Using Equilibrium Propagation

School of Electrical Engineering and Computer Science(电气工程与计算机科学学院);The Pennsylvania State University(宾夕法尼亚州立大学)

Malyaban Bal, Abhronil Sengupta

AI总结:

针对平衡传播(EP)无法处理动态序列输入的局限,研究将现代Hopfield网络作为注意力机制与EP的收敛型RNN集成,首次将EP应用于IMDB情感分析和SNLI自然语言推理任务。

摘要:

平衡传播(Equilibrium Propagation,EP)是一种强大且更具生物合理性的替代方案,可取代反向传播等传统学习框架。EP的有效性源于它仅依赖局部计算,且在两个训练阶段都只需要一种计算单元,因此在类脑神经形态计算等领域具有更高的适用性。EP中模型的动力学由能量函数控制,模型内部状态会根据该能量函数定义的状态转移规则收敛到稳态。但根据定义,EP要求模型(一种收敛型循环神经网络RNN)的输入在两个训练阶段都保持静态,因此无法使用EP设计具备LSTM或GRU类架构的序列分类模型。本文利用现代Hopfield网络的最新进展,进一步理解基于能量的模型,并在满足EP收敛准则、保持其与循环反向传播理论相似性的前提下,开发了适用于复杂序列分类任务的EP解决方案。我们探索了将现代Hopfield网络作为注意力机制,与EP中使用的收敛型RNN模型集成的可能性,从而首次将EP的适用范围扩展到自然语言处理中的两类不同序列分类任务,即情感分析(IMDB数据集)和自然语言推理(SNLI数据集)。

英文摘要:

Equilibrium Propagation (EP) is a powerful and more bio-plausible alternative to conventional learning frameworks such as backpropagation. The effectiveness of EP stems from the fact that it relies only on local computations and requires solely one kind of computational unit during both of its training phases, thereby enabling greater applicability in domains such as bio-inspired neuromorphic computing. The dynamics of the model in EP is governed by an energy function and the internal states of the model consequently converge to a steady state following the state transition rules defined by the same. However, by definition, EP requires the input to the model (a convergent RNN) to be static in both the phases of training. Thus it is not possible to design a model for sequence classification using EP with an LSTM or GRU like architecture. In this paper, we leverage recent developments in modern hopfield networks to further understand energy based models and develop solutions for complex sequence classification tasks using EP while satisfying its convergence criteria and maintaining its theoretical similarities with recurrent backpropagation. We explore the possibility of integrating modern hopfield networks as an attention mechanism with convergent RNN models used in EP, thereby extending its applicability for the first time on two different sequence classification tasks in natural language processing viz. sentiment analysis (IMDB dataset) and natural language inference (SNLI dataset).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑