二阶序列机器遗忘中的误差累积
Error Accumulation in Second-Order Sequential Machine Unlearning
AI总结:
本文研究高维下二阶序列机器遗忘的误差累积,分析两种隐私设置与两类认证下的牛顿-CG更新次数,通过实验验证该方法在病态问题上的序列跟踪与计算优势。
摘要:
机器遗忘旨在在不进行完全重新训练的情况下移除已删除训练数据的影响。在序列场景中,删除请求随时间到达,重复更新会累积近似误差。本文研究高维情况下的误差累积,其中参数数量$p$与样本量$n$成正比,针对两种隐私设置下的二阶遗忘方法:(i)部分遗忘,允许保留未发布的内部状态,同时发布模型的认证版本;(ii)完全遗忘,仅保留已发布的模型且不保留任何隐藏状态。我们还区分逐点和逐路径认证,取决于认证保证适用于每个发布的模型还是整个发布序列。我们分析共轭梯度(CG)作为牛顿迭代中逆海森向量积(IHVP)的高效求解器。理论表明,对于逐点部分、逐路径部分、逐点完全和逐路径完全认证,分别最多需要1/2/2/3次牛顿-CG更新即可实现成功认证,且发布模型的总体风险退化消失。数值实验表明,结合共轭梯度的基于牛顿的方法相较于基于梯度的更新,在病态问题上表现出准确的序列跟踪和显著的计算增益。
英文摘要:
Machine unlearning seeks to remove the influence of deleted training data without full retraining. In the sequential setting, deletion requests arrive over time, and repeated updates can accumulate approximation errors. In this paper, we study error accumulation in high dimensions, where parameter count $p$ is proportional to sample size $n$, for second-order unlearning methods under two privacy settings: (i) \emph{partial unlearning}, which allows unpublished internal states to be retained while certified versions of the model are released; and (ii) \emph{perfect unlearning}, which retains only published models and no hidden states. We also distinguish pointwise and pathwise certification, depending on whether the certification guarantee applies to each released model or the whole released sequence. We analyze conjugate gradient (CG) as an efficient solver for the inverse Hessian-vector product (IHVP) in Newton iterations. Our theory shows that a maximum of 1/2/2/3 Newton--CG updates suffice for successful certification and vanishing population-risk degradation of the published model, for pointwise partial, pathwise partial, pointwise perfect and pathwise perfect certification, respectively. Numerical experiments demonstrate accurate sequential tracking and substantial computational gains of Newton-based methods combined with conjugate gradient, over gradient-based updates, especially for ill-conditioned problems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。