Data-driven machine learning models are being increasingly employed in several important inference problems in biology, chemistry, and physics which require learning over combinatorial spaces. Recent empirical evidence (see, e.g., [1], [2], [3]) suggests that regularizing the spectral representation of such models improves their generalization power when labeled data is scarce. However, despite these empirical studies, the theoretical underpinning of when and how spectral regularization enables improved generalization is poorly understood. In this paper, we focus on learning pseudo-Boolean functions and demonstrate that regularizing the empirical mean squared error by the L_1 norm of the spectral transform of the learned function reshapes the loss landscape and allows for data-frugal learning, under a restricted secant condition on the learner's empirical error measured against the ground truth function. Under a weaker quadratic growth condition, we show that stationary points which also approximately interpolate the training data points achieve statistically optimal generalization performance. Complementing our theory, we empirically demonstrate that running gradient descent on the regularized loss results in a better generalization performance compared to baseline algorithms in several data-scarce real-world problems.
Kernel Clustering with Sigmoid-based Regularization for Efficient Segmentation of Sequential Data
Hanoi University of Science and Technology（河内理工大学）；The Graduate University for Advanced Studies, SOKENDAI（综合研究大学院大学（SOKENDAI））；National Institute of Informatics（国立信息学研究所）
Implicit Binarization via Complex Phase Dynamics in Combinatorial Optimization
School of Physics and Astronomy, Tel Aviv University（特拉维夫大学物理与天文学学院）；School of Electrical and Computer Engineering, Tel Aviv University（特拉维夫大学电气与计算机工程学院）
Department of Computer Systems, Universidad Politécnica de Madrid（马德里理工大学计算机系统系）；Department of Applied Mathematics, Universidad Politécnica de Madrid（马德里理工大学应用数学系）
Variational Annealing on Graphs for Combinatorial Optimization
ELLIS Unit Linz（欧洲学习与智能系统实验室林茨分部）；LIT AI Lab（LIT人工智能实验室）；Institute for Machine Learning, Johannes Kepler University Linz（林茨约翰内斯·开普勒大学机器学习研究所）；Johannes Kepler University Linz（林茨约翰内斯·开普勒大学）
Optimization Networks for Integrated Machine Learning
Johannes Kepler University Linz（约翰内斯开普勒大学林茨分校）；University of Applied Sciences Upper Austria（上奥地利州应用科学大学）；Institute for Formal Models and Verification（形式模型与验证研究所）；Heuristic and Evolutionary Algorithms Laboratory（启发式与进化算法实验室）
Deep ReLU Networks Have Surprisingly Simple Polytopes
The Chinese University of Hong Kong（香港中文大学）；RIKEN Center for Advanced Intelligence Project (AIP)（理化学研究所先进智能项目中心）；Peking University（北京大学）；Harbin Institute of Technology（哈尔滨工业大学）；Weill Cornell Medicine, Cornell University（康奈尔大学威尔医学院）
Regularization vs. Relaxation: A conic optimization perspective of statistical variable selection
Department of Mathematics, Washington State University（数学系，华盛顿州立大学）；Department of Statistics, University of Connecticut（统计系，康涅狄格大学）；Department of Industrial and Systems Engineering, University of Wisconsin-Madison（工业与系统工程系，威斯康星大学麦迪逊分校）
