Due to diverse architectures in deep neural networks (DNNs) with severe overparameterization, regularization techniques are critical for finding optimal solutions in the huge hypothesis space. In this paper, we propose an effective regularization technique, called Neighborhood Region Smoothing (NRS). NRS leverages the finding that models would benefit from converging to flat minima, and tries to regularize the neighborhood region in weight space to yield approximate outputs. Specifically, gap between outputs of models in the neighborhood region is gauged by a defined metric based on Kullback-Leibler divergence. This metric provides similar insights with the minimum description length principle on interpreting flat minima. By minimizing both this divergence and empirical loss, NRS could explicitly drive the optimizer towards converging to flat minima. We confirm the effectiveness of NRS by performing image classification tasks across a wide range of model architectures on commonly-used datasets such as CIFAR and ImageNet, where generalization ability could be universally improved. Also, we empirically show that the minima found by NRS would have relatively smaller Hessian eigenvalues compared to the conventional method, which is considered as the evidence of flat minima.
Department of Statistics and Bioinformatics and Computational Biology Program（统计学与生物信息学及计算生物学项目部）；School of Mathematical Sciences and Center for Statistical Science（数学科学学院和统计科学中心）
When predict can also explain: few-shot prediction to select better neural latents
Faculty of Mathematics, Technion – Israel Institute of Technology（数学系，技术离子理工学院）；Rappaport Faculty of Medicine and Network Biology Research Laboratory, Technion - Israel Institute of Technology（医学与网络生物学研究实验室，技术离子理工学院）
Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization
Department of Statistics, LMU Munich（慕尼黑大学统计系）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Institute of Computational Biology, Helmholtz Munich（亥姆霍兹慕尼黑计算生物学研究所）；Center for Computational Mathematics, Flatiron Institute（熨斗研究所计算数学中心）
HwaMei Hospital, University of Chinese Academy of Sciences（中国科学院大学华美医院）；Ningbo Institute of Life and Health Industry, University of Chinese Academy of Sciences（中国科学院大学宁波生命与健康产业研究院）
