Support vector classification (SVC) is a classical and well-performed learning method for classification problems. A regularization parameter, which significantly affects the classification performance, has to be chosen and this is usually done by the cross-validation procedure. In this paper, we reformulate the hyperparameter selection problem for support vector classification as a bilevel optimization problem in which the upper-level problem minimizes the average number of misclassified data points over all the cross-validation folds, and the lower-level problems are the l1-loss SVC problems, with each one for each fold in T-fold cross-validation. The resulting bilevel optimization model is then converted to a mathematical program with equilibrium constraints (MPEC). To solve this MPEC, we propose a global relaxation cross-validation algorithm (GR-CV) based on the well-know Sholtes-type global relaxation method (GRM). It is proven to converge to a C-stationary point. Moreover, we prove that the MPEC-tailored version of the Mangasarian-Fromovitz constraint qualification (MFCQ), which is a key property to guarantee the convergence of the GRM, automatically holds at each feasible point of this MPEC. Extensive numerical results verify the efficiency of the proposed approach. In particular, compared with other methods, our algorithm enjoys superior generalization performance over almost all the data sets used in this paper.
Bilevel Programming Approach for Image Restoration Problems with Automatically Hyperparameter Selection
School of Mathematics and Statistics, Henan University（河南大学数学与统计学院）；Center for Applied Mathematics of Henan Province, Henan University（河南省应用数学中心，河南大学）；School of Nursing and Health, Henan University（河南大学护理与健康学院）；School of Software, Henan University（河南大学软件学院）
On Constraint Qualifications for MPECs with Applications to Bilevel Hyperparameter Optimization for Machine Learning
School of Mathematics and Statistics, Beijing Institute of Technology（北京理工大学数学与统计学院）；Beijing Key Laboratory on MCAACI/Key Laboratory of Mathematical Theory and Computation in Information Security, Beijing Institute of Technology（北京信息计算安全数学理论与计算重点实验室）；School of Mathematical Sciences, University of Southampton（南安普顿大学数学科学学院）
Value Function Based Difference-of-Convex Algorithm for Bilevel Hyperparameter Selection Problems
University of Waterloo（滑铁卢大学）；University of Victoria（维多利亚大学）；Southern University of Science and Technology（南方科技大学）；National Center for Applied Mathematics Shenzhen（深圳国家应用数学中心）
Alternating Gradient-Type Algorithm for Bilevel Optimization with Inexact Lower-Level Solutions via Moreau Envelope-based Reformulation
Southern University of Science and Technology（南方科技大学）；National Center for Applied Mathematics Shenzhen（深圳应用数学国家研究中心）；City University of Hong Kong（香港城市大学）
An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning
EPSRC Centre for Doctoral Training in Statistical Applied Mathematics at Bath (SAMBa)（EPSRC统计应用数学博士培训中心（SAMBa））；Department of Mathematical Sciences, University of Bath（巴斯大学数学科学系）；Department of Electronics & Electrical Communication Engineering, Indian Institute of Technology (IIT), Kharagpur（印度理工学院（Kharagpur）电子与电气通信工程系）；School of Mathematics and Statistics, University of Sydney（悉尼大学数学与统计学学院）
FAB: A First-Order AB-based Gradient Algorithm for Distributed Bilevel Optimization over Time-Varying Directed Graphs
Southern University of Science and Technology（南方科技大学）；Peng Cheng Laboratory（鹏城实验室）；School of Computer Science and Engineering, Sun Yat-Sen University（中山大学计算机科学与工程学院）；School of Artificial Intelligence, Southern University of Science and Technology（南方科技大学人工智能学院）；National Center for Applied Mathematics Shenzhen（深圳应用数学国家研究中心）；Detection Institute for Advanced Technology Longhua-Shenzhen（深圳龙华先进技术检测研究院）
SUN-DSBO: A Structured Unified Framework for Nonconvex Decentralized Stochastic Bilevel Optimization
Southern University of Science and Technology（南方科技大学）；Pengcheng Laboratory（鹏城实验室）；Sun Yat-sen University（中山大学）；National Center for Applied Mathematics Shenzhen（深圳应用数学国家中心）
Moreau Envelope Based Difference-of-weakly-Convex Reformulation and Algorithm for Bilevel Programs
University of British Columbia（不列颠哥伦比亚大学）；University of Victoria（维多利亚大学）；Southern University of Science and Technology（南方科技大学）；National Center for Applied Mathematics Shenzhen（深圳国家应用数学中心）；Peng Cheng Laboratory（鹏城实验室）
S$^3$LDBO: A Snapshot Single-Loop Algorithm for Decentralized Bilevel Optimization
School of Mathematics, Hohai University（河海大学数学学院）；School of Mathematics, Nanjing University（南京大学数学学院）；Department of Computational Applied Mathematics and Operations Research, Rice University（理海大学计算应用数学与运筹学系）
A Hybrid Subgradient Method for Nonsmooth Nonconvex Bilevel Optimization
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen University（深圳大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）
Bilevel Learning with Inexact Stochastic Gradients
Department of Mathematical Sciences, University of Bath（巴斯大学数学科学系）；Department of Electronics & Electrical Communication Engineering, Indian Institute of Technology (IIT) Kharagpur（印度理工学院Kharagpur电子与电气通信工程系）；School of Mathematics and Statistics, University of Sydney（悉尼大学数学与统计学学院）
