词的相互依赖揭示了LSTM如何构成表示
Word Interdependence Exposes How LSTMs Compose Representations
摘要：
NLP最近的工作表明，LSTM语言模型可以捕获语言数据中的组合结构。为了更仔细地了解这些表示是如何分层组成的，我们提出了一种新的LSTM中词义之间相互依赖的度量方法，基于它们在内部门的相互作用。为了探索组合表征是如何在训练过程中出现的，我们对合成数据进行了简单的实验，通过显示高度的相互依赖性会如何损害泛化来说明我们的测量。这些合成实验还说明了一个关于在训练过程中如何发现层次结构的特定假设：父母成分依赖于他们孩子的有效表征，而不是独立学习远程关系。我们进一步支持这一措施与英语语言数据的实验，其中的相互依存性更高的句法连接的单词对。
英文摘要：
Recent work in NLP shows that LSTM language models capture compositional structure in language data. For a closer look at how these representations are composed hierarchically, we present a novel measure of interdependence between word meanings in an LSTM, based on their interactions at the internal gates. To explore how compositional representations arise over training, we conduct simple experiments on synthetic data, which illustrate our measure by showing how high interdependence can hurt generalization. These synthetic experiments also illustrate a specific hypothesis about how hierarchical structures are discovered over the course of training: that parent constituents rely on effective representations of their children, rather than on learning long-range relations independently. We further support this measure with experiments on English language data, where interdependence is higher for more closely syntactically linked word pairs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
