arXiv 2605.126482026-09-30cs.LGstat.ML

针对DP-SGD与相关噪声的Kolmogorov-Arnold网络的总体风险界

Optimization Risk Bounds for Kolmogorov-Arnold Networks Trained by DP-SGD with Correlated Noise

RPTU Kaiserslautern-Landau（凯撒斯劳滕-兰道大学）；Machine Learning Research, Morgan Stanley（摩根大通机器学习研究部）；Institute of Science and Technology, Klosterneuburg（克洛斯特纽堡科学与技术研究所）；Catholic University of Eichstätt-Ingolstadt（埃希施泰特-英戈尔施塔特天主教大学）

Puyu Wang, Jan Schuchardt, Nikita Kalinin, Marcio Monteiro, Junyu Zhou, Sophie Fellenz, Christoph Lampert, Marius Kloft

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文为通过DP-SGD与相关噪声训练的KANs建立了首个总体风险界，涵盖非隐私SGD及DP-SGD，改进了KAN理论并优化了隐私-效用权衡。

摘要：

我们为通过小批量SGD和梯度裁剪训练的Kolmogorov-Arnold网络（KANs）建立了首个总体风险界，涵盖了非隐私SGD以及具有Gaussian扰动的差分隐私SGD（DP-SGD），其扰动在独立和时间相关噪声之间插值。此设置在两个轴上比之前的KAN理论更接近实际应用：训练使用小批量SGD，这是现代网络的标准配方，而不是全批量梯度下降（GD）；且时间相关噪声机制在经验上显示出比独立噪声机制更有利的隐私-效用权衡。我们的结果涵盖了Wang等人（2026）为KANs对应的全批量GD和独立噪声DP-GD结果，同时提供了更尖锐的固定第二层特化。技术核心是针对非凸情况下的相关噪声DP训练的新分析路径。时间依赖性破坏了标准一步SGD论证所依赖的条件中心化结构，而投影步骤阻碍了相关扰动的精确抵消结构。我们通过辅助未投影动态、一个吸收当前噪声扰动的移位迭代以及高概率Bootstrap证明投影不活动，来解决这些困难。将此优化分析与基于稳定性的泛化论证相结合，获得了所述的总体风险界。据我们所知，这是首次对DP训练中相关噪声机制的优化和总体风险分析，超越了凸学习，特别是针对神经网络。

英文摘要：

The theoretical understanding of differentially private stochastic gradient descent (DP-SGD) with temporally correlated noise remains limited, particularly for non-convex neural network training. As a first step, we study two-layer Kolmogorov-Arnold Networks (KANs), a recently introduced architecture with learnable spline-based edge functions. We establish the first optimization risk bounds for clipped mini-batch DP-SGD with correlated noise in this setting, with explicit dependence on temporal correlation, clipping, mini-batch sampling, and network width. Existing arguments fail for three reasons: temporal dependence breaks the conditional-centering step; projection obstructs the cross-iteration cancellation of correlated perturbations; and active clipping breaks the empirical-gradient structure. We address these issues through shifted and auxiliary dynamics, a weighted empirical loss, and a high-probability localization argument. Our bound shows that temporal correlation reduces the leading noise terms, the clipping threshold enters essentially through an effective step size, and private training admits an explicit network width range. Experiments on synthetic data and MNIST support the predicted optimization effect of temporal correlation. As an application, we derive population risk guarantees via algorithmic stability. Our framework recovers non-private mini-batch SGD, independent-noise DP-SGD, and their full-batch counterparts as special cases.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。