In this paper, we consider an $\ell_{0}$-norm penalized formulation of the generalized eigenvalue problem (GEP), aimed at extracting the leading sparse generalized eigenvector of a matrix pair. The formulation involves maximization of a discontinuous nonconcave objective function over a nonconvex constraint set, and is therefore computationally intractable. To tackle the problem, we first approximate the $\ell_{0}$-norm by a continuous surrogate function. Then an algorithm is developed via iteratively majorizing the surrogate function by a quadratic separable function, which at each iteration reduces to a regular generalized eigenvalue problem. A preconditioned steepest ascent algorithm for finding the leading generalized eigenvector is provided. A systematic way based on smoothing is proposed to deal with the "singularity issue" that arises when a quadratic function is used to majorize the nondifferentiable surrogate function. For sparse GEPs with special structure, algorithms that admit a closed-form solution at every iteration are derived. Numerical experiments show that the proposed algorithms match or outperform existing algorithms in terms of computational complexity and support recovery.
Relaxed Sparse Eigenvalue Conditions for Sparse Estimation via Non-convex Regularized Regression
Department of Automation, Tsinghua University, Beijing 100084, P.R.China(自动化系,清华大学,北京100084,中国);State Key Lab of Intelligent Technologies and Systems(智能技术与系统国家重点实验室);Tsinghua National Laboratory for Information Science and Technology(TNList)(信息科学与技术国家实验室(清华大学))
Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization
Department of Statistics, LMU Munich(慕尼黑大学统计系);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Institute of Computational Biology, Helmholtz Munich(亥姆霍兹慕尼黑计算生物学研究所);Center for Computational Mathematics, Flatiron Institute(熨斗研究所计算数学中心)