arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1502.02478cs.NEcs.CV
收藏

使用子矩阵的高效分批丢弃训练

Efficient batchwise dropout training using submatrices

Ben Graham, Jeremy Reizenstein, Leigh Robinson

摘要:

丢弃是一种用于正则化人工神经网络的流行技术。Dropout网络通常通过minibatch梯度下降进行训练,其中dropout掩码关闭某些单元--对minibatch中的每个样本应用不同的dropout模式。我们探索一个非常简单的替代dropout掩码。而不是通过将它们设置为零来屏蔽丢弃的单元,我们使用权重矩阵的子矩阵执行矩阵乘法--不需要的隐藏单元永远不会计算。执行dropout batchwise,这样一种dropout模式用于minibatch中的每个样本,我们可以大大减少训练时间。分批丢弃可以与全连接和卷积神经网络一起使用。

英文摘要:

Dropout is a popular technique for regularizing artificial neural networks. Dropout networks are generally trained by minibatch gradient descent with a dropout mask turning off some of the units---a different pattern of dropout is applied to every sample in the minibatch. We explore a very simple alternative to the dropout mask. Instead of masking dropped out units by setting them to zero, we perform matrix multiplication using a submatrix of the weight matrix---unneeded hidden units are never calculated. Performing dropout batchwise, so that one pattern of dropout is used for each sample in a minibatch, we can substantially reduce training times. Batchwise dropout can be used with fully-connected and convolutional neural networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑