Domain knowledge is useful to improve the generalization performance of learning machines. Sign constraints are a handy representation to combine domain knowledge with learning machine. In this paper, we consider constraining the signs of the weight coefficients in learning the linear support vector machine, and develop two optimization algorithms for minimizing the empirical risk under the sign constraints. One of the two algorithms is based on the projected gradient method, in which each iteration of the projected gradient method takes $O(nd)$ computational cost and the sublinear convergence of the objective error is guaranteed. The second algorithm is based on the Frank-Wolfe method that also converges sublinearly and possesses a clear termination criterion. We show that each iteration of the Frank-Wolfe also requires $O(nd)$ cost. Furthermore, we derive the explicit expression for the minimal iteration number to ensure an $ε$-accurate solution by analyzing the curvature of the objective function. Finally, we empirically demonstrate that the sign constraints are a promising technique when similarities to the training examples compose the feature vector.
Efficient Relaxed Gradient Support Pursuit for Sparsity Constrained Non-convex Optimization
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education(教育部智能感知与图像理解重点实验室);School of Artificial Intelligence, Xidian University(西安电子科技大学人工智能学院);Department of Computer Science, The University of Texas at Austin(德克萨斯大学奥斯汀分校计算机科学系)
Constrained Hebbian Learning Supports Efficient Representational Allocation under Structural Constraints
KEIM Institute, Albstadt-Sigmaringen University(凯姆研究所,阿尔布施塔特 - 西格马林根大学);Department of Computer Science, Chemnitz University of Technology(计算机科学系,开姆尼茨工业大学)
Iterated Support Vector Machines for Distance Metric Learning
Harbin Institute of Technology(哈尔滨工业大学);The Hong Kong Polytechnic University(香港理工大学);Sun Yat-Sen University(中山大学);NEC Laboratories China(NEC中国研究院);Xi’an Jiaotong University(西安交通大学)