arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2112.09174cs.CL
收藏

用LSTM和Transformer学习有界上下文无关文法:区别和解释

Learning Bounded Context-Free-Grammar via LSTM and the Transformer:Difference and Explanations

University of California San Diego(加州大学圣地亚哥分校);Facebook AI Research(Facebook人工智能研究院);University of California, Berkeley(加州大学伯克利分校)

Hui Shi, Sicun Gao, Yuandong Tian, Xinyun Chen, Jishen Zhao

摘要:

长短期记忆(LSTM)和Transformers是两种用于自然语言处理任务的流行神经架构。理论结果表明,两者都是图灵完备的,可以表示任何上下文无关语言(CFL)。在实践中,经常观察到Transformer模型比LSTM具有更好的表示能力。但其中的原因却鲜为人知。我们研究了LSTM和Transformer之间的实际差异,并基于它们的潜在空间分解模式提出了一种解释。为了实现这一目标,我们引入了一个oracle训练范式,它强制分解LSTM和Transformer的潜在表示,并监督相应CFL的下推自动机(PDA)的转换。通过强制分解,我们证明了LSTM和Transformer在学习CFL方面的性能上限是接近的:它们都可以模拟堆栈并执行堆栈操作以及状态转换。然而,缺乏强制分解导致LSTM模型无法捕获堆栈和堆栈操作,同时对Transformer模型产生轻微影响。最后,我们将PDA原型上的实验与真实的句法分析任务相结合,再次验证了本文的结论

英文摘要:

Long Short-Term Memory (LSTM) and Transformers are two popular neural architectures used for natural language processing tasks. Theoretical results show that both are Turing-complete and can represent any context-free language (CFL).In practice, it is often observed that Transformer models have better representation power than LSTM. But the reason is barely understood. We study such practical differences between LSTM and Transformer and propose an explanation based on their latent space decomposition patterns. To achieve this goal, we introduce an oracle training paradigm, which forces the decomposition of the latent representation of LSTM and the Transformer and supervises with the transitions of the Pushdown Automaton (PDA) of the corresponding CFL. With the forced decomposition, we show that the performance upper bounds of LSTM and Transformer in learning CFL are close: both of them can simulate a stack and perform stack operation along with state transitions. However, the absence of forced decomposition leads to the failure of LSTM models to capture the stack and stack operations, while having a marginal impact on the Transformer model. Lastly, we connect the experiment on the prototypical PDA to a real-world parsing task to re-verify the conclusions

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑