Cross-validated feature selection using least-squares support vector machines for binary classification
Graduate School of Science and Technology, University of Tsukuba(筑波大学科学技术研究院);Institute of Systems and Information Engineering, University of Tsukuba(筑波大学系统信息工程学院);Gurobi Japan Co., Ltd.(Gurobi日本有限公司);Institute of Engineering, Tokyo University of Agriculture and Technology(东京农工大学工学院)
Support vector machines (SVMs) are widely used for binary classification. However, irrelevant or redundant features can degrade classification accuracy and model interpretability, making feature selection important for SVMs. Most SVM feature selection methods rely on sequential search or regularization and do not directly identify the best feature subset. Mixed-integer optimization (MIO) is a direct approach to best subset selection. Cross-validation (CV) is broadly applicable to feature selection because it evaluates predictive performance on validation data without model assumptions. Although Purcaru et al. (2026) proposed an MIO formulation for CV-based linear-SVM feature selection (SVM-CV), this method suffers from high computational cost. We develop computationally efficient CV-based MIO formulations for feature selection in linear SVM classification by using least squares support vector machines (LS-SVMs). We formulate the problem as a bilevel MIO problem, where the lower-level problem trains LS-SVM classifiers and the upper-level problem selects features based on validation sets. We then reformulate it as a single-level MIO problem by exploiting the optimality conditions of LS-SVM. We propose two formulations: a hinge-loss-based validation model and a least-squares-based validation model. We compare our methods with SVM-CV and other SVM-based and logistic-regression-based methods through simulation experiments on synthetic data. The hinge-loss-based validation model attained high feature selection accuracy under the high-noise and moderate-noise conditions, particularly with more candidate features, and the least-squares-based validation model performed well in feature selection under the low-noise condition at the lower correlation level. Furthermore, our two models achieved substantially faster computation than the existing SVM-CV model.