arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1903.07435cs.CL
收藏

LSTM语言模型中的数和语法单位的出现

The emergence of number and syntax units in LSTM language models

Cognitive Neuroimaging Unit(认知神经影像研究组);NeuroSpin center(神经自旋中心);Facebook AI Research(Facebook人工智能研究院);ILLC, University of Amsterdam(阿姆斯特丹大学语言、逻辑与计算研究所);Paris(巴黎(此处按要求仅保留机构相关部分,原文本中“Paris, France”作为机构的补充信息,机构本身为Facebook AI Research));Gif-sur-Yvette, France((地址信息,非机构,按要求忽略));Amsterdam, Netherlands((地址信息,非机构,按要求忽略))

Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, Marco Baroni

摘要:

最近的工作表明,在通用语言建模目标上训练的LSTM可以捕获语法敏感的泛化,例如长距离数字一致性。然而,我们对他们如何完成这一非凡壮举没有机械的理解。有些人认为这取决于没有真正考虑到层次结构的竞争力。在这里,我们提出了一个详细的研究在单个神经元水平的LSTM的内部机制的数字跟踪。我们发现,长途号码信息主要由两个“号码单元”管理。重要的是,这些单位的行为部分地由其他单位独立地控制,以跟踪句法结构。我们的结论是,LSTMs,在某种程度上,真正实现句法处理机制,铺平了道路,更普遍的理解语法编码LSTMs。

英文摘要:

Recent work has shown that LSTMs trained on a generic language modeling objective capture syntax-sensitive generalizations such as long-distance number agreement. We have however no mechanistic understanding of how they accomplish this remarkable feat. Some have conjectured it depends on heuristics that do not truly take hierarchical structure into account. We present here a detailed study of the inner mechanics of number tracking in LSTMs at the single neuron level. We discover that long-distance number information is largely managed by two `number units'. Importantly, the behaviour of these units is partially controlled by other units independently shown to track syntactic structure. We conclude that LSTMs are, to some extent, implementing genuinely syntactic processing mechanisms, paving the way to a more general understanding of grammatical encoding in LSTMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑