arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1906.01634cs.CLcs.AIcs.LG
收藏

神经网络中组合性的实现

On the Realization of Compositionality in Neural Networks

University of Amsterdam(阿姆斯特丹大学);University of Tübingen(图宾根大学);Universitat Pompeu Fabra(庞培法布拉大学)

Joris Baan, Jana Leible, Mitja Nikolaus, David Rau, Dennis Ulmer, Tim Baumgärtner, Dieuwke Hupkes, Elia Bruni

摘要:

我们提出了一个详细的比较两种类型的序列序列模型训练进行合成任务。这些模型在推理时在架构上是相同的,但它们的训练方式不同:我们的基线模型仅用任务成功信号训练,而另一个模型在其注意力机制(注意力引导)上接受额外的监督,这已被证明是鼓励更多组合解决方案的有效方法(Hupkes et al.,2019年)。我们首先确认,通过在Liška et al.(2019)提出的查找表任务上对模型进行训练,具有细心指导的模型确实比基线推断出更多的组合解决方案。然后,我们对两种模型类型之间的结构差异进行了深入分析,特别关注参数空间和隐藏层激活的组织,并在这两个方面发现了明显的差异。引导网络更多地关注输入的组成部分,而不是整个序列,并开发出具有特定目的的小功能神经元组,这些功能神经元组更有选择性地使用它们的门。参数热图、组件交换和图形分析的结果也表明,引导网络表现出更模块化的结构,具有少量专门的、强连接的神经元。

英文摘要:

We present a detailed comparison of two types of sequence to sequence models trained to conduct a compositional task. The models are architecturally identical at inference time, but differ in the way that they are trained: our baseline model is trained with a task-success signal only, while the other model receives additional supervision on its attention mechanism (Attentive Guidance), which has shown to be an effective method for encouraging more compositional solutions (Hupkes et al.,2019). We first confirm that the models with attentive guidance indeed infer more compositional solutions than the baseline, by training them on the lookup table task presented by Liška et al. (2019). We then do an in-depth analysis of the structural differences between the two model types, focusing in particular on the organisation of the parameter space and the hidden layer activations and find noticeable differences in both these aspects. Guided networks focus more on the components of the input rather than the sequence as a whole and develop small functional groups of neurons with specific purposes that use their gates more selectively. Results from parameter heat maps, component swapping and graph analysis also indicate that guided networks exhibit a more modular structure with a small number of specialized, strongly connected neurons.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑