arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.10745cs.AIcs.CLcs.LG
收藏

通过模块化和扩充提高系统的泛化能力

Improving Systematic Generalization Through Modularity and Augmentation

University College London(伦敦大学学院);New York University(纽约大学);Facebook AI Research(脸书人工智能研究院)

Laura Ruis, Brenden Lake

摘要:

系统泛化是将已知部分组合成新意义的能力;这是人类有效学习的一个重要方面,但却是神经网络学习的一个弱点。在这项工作中,我们研究了两个著名的建模原则-模块化和数据增强-如何影响扎根语言学习中神经网络的系统泛化。我们分析了需要多大的词汇量来实现系统的泛化,以及增强数据需要与手头的问题有多相似。我们的研究结果表明,即使在一个合成基准的控制设置,实现系统的泛化仍然非常困难。在使用比原始问题多近40倍的副词的增强数据集进行训练后,非模块化基线无法系统地推广到已知动词和副词的新颖组合。当将任务分离为感知和导航等认知过程时,模块化神经网络能够利用增强的数据并更系统地进行概括,在两个先前未改进的gSCAN测试上实现了70%和40%的精确匹配。我们希望这项工作能让我们深入了解系统泛化的驱动因素,以及我们仍然需要改进的神经网络,以便像人类一样学习。

英文摘要:

Systematic generalization is the ability to combine known parts into novel meaning; an important aspect of efficient human learning, but a weakness of neural network learning. In this work, we investigate how two well-known modeling principles -- modularity and data augmentation -- affect systematic generalization of neural networks in grounded language learning. We analyze how large the vocabulary needs to be to achieve systematic generalization and how similar the augmented data needs to be to the problem at hand. Our findings show that even in the controlled setting of a synthetic benchmark, achieving systematic generalization remains very difficult. After training on an augmented dataset with almost forty times more adverbs than the original problem, a non-modular baseline is not able to systematically generalize to a novel combination of a known verb and adverb. When separating the task into cognitive processes like perception and navigation, a modular neural network is able to utilize the augmented data and generalize more systematically, achieving 70% and 40% exact match increase over state-of-the-art on two gSCAN tests that have not previously been improved. We hope that this work gives insight into the drivers of systematic generalization, and what we still need to improve for neural networks to learn more like humans do.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑