arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.13398cs.LGcs.CR
收藏

展开SGD:了解影响机器遗忘的因素

Unrolling SGD: Understanding Factors Influencing Machine Unlearning

University of Toronto(多伦多大学);Vector Institute(向量研究所);University of Wisconsin-Madison(威斯康星大学麦迪逊分校)

Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, Nicolas Papernot

摘要:

机器遗忘是使部署的机器学习模型忘记其某些训练数据点的过程。虽然从零开始对模型进行简单的重新训练是一种选择,但对于深度学习模型来说,它几乎总是与巨大的计算开销相关联。因此,已经提出了几种近似遗忘的方法,以及相应的度量标准,这些度量标准形式化了模型忘记数据点意味着什么。在这项工作中,我们首先对近似非学习的方法和度量进行分类。因此,我们识别验证错误,即,近似未学习模型和单纯再训练模型的权重之间的L2差,作为近似未学习度量,因为它包含了一大类其他度量,所以应该对其进行优化。我们从理论上分析了经典的训练算法,随机梯度下降(SGD),表面的变量,这是有关减少近似unlearning SGD的验证误差。从这个分析中,我们首先得到一个易于计算的代理验证错误(称为unlearning错误)。该分析还为新的训练目标惩罚的设计提供了信息,该惩罚限制了SGD期间权重的总体变化,从而有助于以较低的验证误差近似遗忘。我们通过CIFAR-10、CIFAR-100和IMDB情感分析对学习的实证评估来验证我们的理论工作。

英文摘要:

Machine unlearning is the process through which a deployed machine learning model is made to forget about some of its training data points. While naively retraining the model from scratch is an option, it is almost always associated with large computational overheads for deep learning models. Thus, several approaches to approximately unlearn have been proposed along with corresponding metrics that formalize what it means for a model to forget about a data point. In this work, we first taxonomize approaches and metrics of approximate unlearning. As a result, we identify verification error, i.e., the L2 difference between the weights of an approximately unlearned and a naively retrained model, as an approximate unlearning metric that should be optimized for as it subsumes a large class of other metrics. We theoretically analyze the canonical training algorithm, stochastic gradient descent (SGD), to surface the variables which are relevant to reducing the verification error of approximate unlearning for SGD. From this analysis, we first derive an easy-to-compute proxy for verification error (termed unlearning error). The analysis also informs the design of a new training objective penalty that limits the overall change in weights during SGD and as a result facilitates approximate unlearning with lower verification error. We validate our theoretical work through an empirical evaluation on learning with CIFAR-10, CIFAR-100, and IMDB sentiment analysis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑