arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1909.08975cs.CLcs.AIstat.ML
收藏

神经语言模型分析:上下文分解揭示数字和性别分配中的默认推理

Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment

University of Amsterdam(阿姆斯特丹大学)

Jaap Jumelet, Willem Zuidema, Dieuwke Hupkes

摘要:

最近的大量研究表明,递归神经语言模型能够处理广泛的语法现象。然而,这些模型如何能够如此出色地完成这些非凡的壮举,仍然是一个悬而未决的问题。为了更深入地了解LSTM基于什么样的信息做出决策,我们提出了一个概括的上下文分解(GCD)。特别是,这种设置使我们能够准确地判断预测的哪一部分源于语义分析,哪一部分真正源于句法线索,哪一部分源于模型偏见本身。我们调查这项技术的任务有关的句法协议和共指决议,并发现该模型强烈依赖于一个默认的推理效果来执行这些任务。

英文摘要:

Extensive research has recently shown that recurrent neural language models are able to process a wide range of grammatical phenomena. How these models are able to perform these remarkable feats so well, however, is still an open question. To gain more insight into what information LSTMs base their decisions on, we propose a generalisation of Contextual Decomposition (GCD). In particular, this setup enables us to accurately distil which part of a prediction stems from semantic heuristics, which part truly emanates from syntactic cues and which part arise from the model biases themselves instead. We investigate this technique on tasks pertaining to syntactic agreement and co-reference resolution and discover that the model strongly relies on a default reasoning effect to perform these tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑