While deep learning models and techniques have achieved great empirical success, our understanding of the source of success in many aspects remains very limited. In an attempt to bridge the gap, we investigate the decision boundary of a production deep learning architecture with weak assumptions on both the training data and the model. We demonstrate, both theoretically and empirically, that the last weight layer of a neural network converges to a linear SVM trained on the output of the last hidden layer, for both the binary case and the multi-class case with the commonly used cross-entropy loss. Furthermore, we show empirically that training a neural network as a whole, instead of only fine-tuning the last weight layer, may result in better bias constant for the last weight layer, which is important for generalization. In addition to facilitating the understanding of deep learning, our result can be helpful for solving a broad range of practical problems of deep learning, such as catastrophic forgetting and adversarial attacking. The experiment codes are available at https://github.com/lykaust15/NN_decision_boundary
Decision Boundary-aware Generation for Long-tailed Learning
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China(中国教育部多媒体可信感知与高效计算重点实验室);Xiamen University(厦门大学);College of Computer Science and Software Engineering(计算机科学与软件工程学院);Shenzhen University(深圳大学);Department of Statistical Science(统计科学系);University College London(伦敦大学学院);Division of Arts and Machine Creativity(艺术与机器创造力 division);Hong Kong University of Science and Technology(香港科技大学)
Deep Learning on Retina Images as Screening Tool for Diagnostic Decision Support
Universidad Autónoma de Occidente(奥克西登特自治大学);School of Business and Engineering Vaud (HEIG-VD)(沃州商学院与工程学院(HEIG-VD));University of Applied Sciences of Western Switzerland (HES-SO)(瑞士西部应用科学大学(HES-SO))