Stochastic First-Order (SFO) methods have been a cornerstone in addressing a broad spectrum of modern machine learning (ML) challenges. However, their efficacy is increasingly questioned, especially in large-scale applications where empirical evidence indicates potential performance limitations. In response, this paper proposes an innovative method specifically designed for kernel support vector machines (SVMs). This method not only achieves faster convergence per iteration but also exhibits enhanced scalability when compared to conventional SFO techniques. Diverging from traditional sample average approximation strategies that typically frame kernel SVM as an 'all-in-one' Quadratic Program (QP), our approach adopts adaptive sampling. This strategy incrementally refines approximation accuracy on an 'as-needed' basis. Crucially, this approach also inspires a decomposition-based algorithm, effectively decomposing parameter selection from error estimation, with the latter being independently determined for each data point. To exploit the quadratic nature of the kernel matrix, we introduce a stochastic conjugate subgradient method. This method preserves many benefits of first-order approaches while adeptly handling both nonlinearity and non-smooth aspects of the SVM problem. Thus, it extends beyond the capabilities of standard SFO algorithms for non-smooth convex optimization. The convergence rate of this novel method is thoroughly analyzed within this paper. Our experimental results demonstrate that the proposed algorithm not only maintains but potentially exceeds the scalability of SFO methods. Moreover, it significantly enhances both speed and accuracy of the optimization process.
CSF: Fixed-outline Floorplanning Based on the Conjugate Subgradient Algorithm Assisted by Q-Learning
School of Mathematics and Statistics（数学与统计学学院）；Department of Basic Science（基础科学系）；School of Information Engineering（信息工程学院）；School of Mathematics and Physics（数学与物理学院）
Eigen-Spike Emergence and Quadratic Equivalents for Conjugate Kernels on Nonlinearly Separable Data
Department of Mathematics ICSI and Department of Statistics（数学系ICSI和统计系）；University of California, San Diego, USA（美国加州大学圣地亚哥分校）；University of California, Berkeley, USA（美国加州大学伯克利分校）；Department of Mathematics ICSI, LBNL and Department of Statistics（数学系ICSI、劳伦斯伯克利国家实验室和统计系）
Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks
Yale University（耶鲁大学）；University of California, San Diego（加利福尼亚大学圣地亚哥分校）；Department of Statistics and Data Science（统计与数据科学系）；Department of Mathematics（数学系）
Instituto de Ciencias Matemáticas (ICMAT)（数学科学研究所（ICMAT））；Centro Superior de Investigaciones Científicas (CSIC)（高级科学研究中心（CSIC））；Universidad Autónoma de Madrid (UAM)（马德里自治大学（UAM））；Universidad Autónoma de Madrid (IIC-UAM)（马德里自治大学（IIC-UAM））
Learning in High-Dimensional Feature Spaces Using ANOVA-Based Fast Matrix-Vector Multiplication
TU Chemnitz（开姆尼茨工业大学）；Department of Mathematics, Chair of Applied Functional Analysis（数学系应用泛函分析讲席）；Department of Mathematics, Chair of Scientific Computing（数学系科学计算讲席）
BLOKS: Scaling Kernel Solvers with Sketched Block Preconditioning
Università degli Studi di Genova（热那亚大学）；CNRS（法国国家科学研究中心）；Istituto di Matematica Applicata e Tecnologie Informatiche “E. Magenes”, CNR（意大利国家研究委员会应用数学与信息技术研究所）；Università degli Studi Milano–Bicocca（米兰比可卡大学）；Istituto Italiano di Tecnologia（意大利技术研究院）
