arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2110.09138cs.LGcs.NE
收藏

状态空间约束可提升可微分神经计算机对未见长度输入序列的泛化能力

State-Space Constraints Can Improve the Generalisation of the Differentiable Neural Computer to Input Sequences With Unseen Length

Area Knowledge Discovery, Know-Center Research GmbH(区域知识发现,Know-Center Research GmbH);Department of Microsystems Engineering, University of Freiburg(微系统工程系,弗赖堡大学);Institute of Machine Learning and Neural Computation, Graz University of Technology(机器学习与神经计算研究所,格拉茨技术大学)

Patrick Ofner, Roman Kern

AI总结:

针对记忆增强神经网络(MANNs)难以泛化到未见长度输入序列的问题,提出状态压缩和正则化约束可微分神经计算机(DNC)状态空间的方法,提升其对分布外长度序列的泛化能力,节省训练资源。

摘要:

记忆增强神经网络(MANNs)能够执行排序等算法任务,但往往难以泛化到训练中未遇到的输入序列长度。我们提出两种约束MANN控制器网络状态空间的方法:状态压缩和状态正则化。通过实验证明,这两种方法均可提升特定类型MANN(即可微分神经计算机DNC)对分布外长度输入序列的泛化能力。受约束的DNC可处理比无约束基线控制器网络长2.3倍的输入序列;值得注意的是,施加的约束无需重新训练即可扩展DNC的记忆矩阵,从而能处理长达10.4倍的输入序列。然而,这些改进并非在所有测试的算法任务中都一致。有趣的是,性能更好的解决方案通常具有高度结构化的状态空间,其特征是状态轨迹呈现出更大的曲率和类循环模式。我们的实验工作表明,状态空间约束可使DNC使用更短的输入序列进行训练,从而节省计算资源,并在获取长序列成本高昂时促进训练。

英文摘要:

Memory-augmented neural networks (MANNs) can perform algorithmic tasks such as sorting. However, they often fail to generalise to input sequence lengths not encountered during training. We introduce two approaches that constrain the state space of the MANN's controller network: state compression and state regularisation. We empirically demonstrated that both approaches can improve generalisation to input sequences of out-of-distribution lengths for a specific type of MANN: the differentiable neural computer (DNC). The constrained DNC could process input sequences that were up to 2.3 times longer than those processed by an unconstrained baseline controller network. Notably, the applied constraints enabled the extension of the DNC's memory matrix without the need for retraining and thus allowed the processing of input sequences that were 10.4 times longer. However, the improvements were not consistent across all tested algorithmic tasks. Interestingly, solutions that performed better often had a highly structured state space, characterised by state trajectories exhibiting increased curvature and loop-like patterns. Our experimental work demonstrates that state-space constraints can enable the training of a DNC using shorter input sequences, thereby saving computational resources and facilitating training when acquiring long sequences is costly.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑