有理递归
Rational Recurrences
摘要:
尽管神经模型在自然语言处理中取得了巨大的经验性成功,但其中许多缺乏经典机器学习方法所具有的强烈直觉。最近,卷积神经网络(CNN)和加权有限状态自动机(WFSA)之间的联系已经被证明,导致了新的解释和见解。在这项工作中,我们证明了一些递归神经网络也与WFSA共享这种连接。我们正式描述这种连接,定义理性递归是经常性的隐藏状态更新功能,可以写为一个有限的WFSA集的前向计算。我们表明,最近的几个神经模型使用理性递归。我们的分析为这些模型提供了一个新的视角,并有助于设计新的神经架构,从WFSA中汲取灵感。我们提出了一个这样的模型,它在语言建模和文本分类方面的表现优于最近的两个基线。我们的研究结果表明,从WFSA等经典模型中转移直觉可以成为设计和理解神经模型的有效方法。
英文摘要:
Despite the tremendous empirical success of neural models in natural language processing, many of them lack the strong intuitions that accompany classical machine learning approaches. Recently, connections have been shown between convolutional neural networks (CNNs) and weighted finite state automata (WFSAs), leading to new interpretations and insights. In this work, we show that some recurrent neural networks also share this connection to WFSAs. We characterize this connection formally, defining rational recurrences to be recurrent hidden state update functions that can be written as the Forward calculation of a finite set of WFSAs. We show that several recent neural models use rational recurrences. Our analysis provides a fresh view of these models and facilitates devising new neural architectures that draw inspiration from WFSAs. We present one such model, which performs better than two recent baselines on language modeling and text classification. Our results demonstrate that transferring intuitions from classical models like WFSAs can be an effective approach to designing and understanding neural models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。