arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1808.05385cs.NEcs.AIcs.LG
收藏

关于深度神经网络的决策边界

On the Decision Boundary of Deep Neural Networks

KAUST(沙特阿拉伯国王科技大学);CEMSE(计算、能源与数学科学与工程学院(CEMSE))

Yu Li, Lizhong Ding, Xin Gao

摘要:

虽然深度学习模型和技术已经取得了巨大的经验成功,但我们对许多方面的成功来源的理解仍然非常有限。为了弥补这一差距,我们研究了生产深度学习架构的决策边界,该架构对训练数据和模型都有较弱的假设。我们证明,理论和经验,神经网络的最后一个权重层收敛到一个线性SVM训练的输出上的最后一个隐藏层,为二进制的情况下和多类的情况下,常用的交叉熵损失。此外,我们的经验表明,训练一个神经网络作为一个整体,而不是只微调最后一个权重层,可能会导致更好的偏置常数的最后一个权重层,这是很重要的泛化。除了促进对深度学习的理解外,我们的结果还有助于解决深度学习的广泛实际问题,例如灾难性遗忘和对抗性攻击。实验代码可在www.example.com上获得

英文摘要:

While deep learning models and techniques have achieved great empirical success, our understanding of the source of success in many aspects remains very limited. In an attempt to bridge the gap, we investigate the decision boundary of a production deep learning architecture with weak assumptions on both the training data and the model. We demonstrate, both theoretically and empirically, that the last weight layer of a neural network converges to a linear SVM trained on the output of the last hidden layer, for both the binary case and the multi-class case with the commonly used cross-entropy loss. Furthermore, we show empirically that training a neural network as a whole, instead of only fine-tuning the last weight layer, may result in better bias constant for the last weight layer, which is important for generalization. In addition to facilitating the understanding of deep learning, our result can be helpful for solving a broad range of practical problems of deep learning, such as catastrophic forgetting and adversarial attacking. The experiment codes are available at https://github.com/lykaust15/NN_decision_boundary

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑