Deep neural networks (DNNs) are often trained on the premise that the complete training data set is provided ahead of time. However, in real-world scenarios, data often arrive in chunks over time. This leads to important considerations about the optimal strategy for training DNNs, such as whether to fine-tune them with each chunk of incoming data (warm-start) or to retrain them from scratch with the entire corpus of data whenever a new chunk is available. While employing the latter for training can be resource-intensive, recent work has pointed out the lack of generalization in warm-start models. Therefore, to strike a balance between efficiency and generalization, we introduce Learn, Unlearn, and Relearn (LURE) an online learning paradigm for DNNs. LURE interchanges between the unlearning phase, which selectively forgets the undesirable information in the model through weight reinitialization in a data-dependent manner, and the relearning phase, which emphasizes learning on generalizable features. We show that our training paradigm provides consistent performance gains across datasets in both classification and few-shot settings. We further show that it leads to more robust and well-calibrated models.
Channel-Level Semantic Perturbations: Unlearnable Examples for Diverse Training Paradigms
School of Information and Communication Engineering, Dalian University of Technology(大连理工大学信息与通信工程学院);School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);School of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
Fast-FedUL: A Training-Free Federated Unlearning with Provable Skew Resilience
Ecole Polytechnique Federale de Lausanne(洛桑联邦理工学院);Hanoi University of Science and Technology(河内科技大学);Griffith University(格里菲斯大学);Humboldt-Universitat zu Berlin(柏林洪堡大学)
The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retraining
City University of Hong Kong(香港城市大学);Nanjing University of Science and Technology(南京理工大学);Monash University(莫纳什大学);The Hong Kong University of Science and Technology(香港科技大学)
在机器学习中,\textit{the right to be forgotten}的出现催生了一个名为\textit{machine unlearning}的范式,该范式使数据持有者能够主动从训练模型中删除其数据。现有的机器遗忘技术集中于集中式训练,其中访问所有持有者的训练数据是服务器进行遗忘过程所必须的。当无法完全访问所有训练数据时,如何实现遗忘,这在很大程度上还有待探索。一个值得注意的例子是联邦学习(FL),其中每个参与的数据持有者在本地进行训练,而不将其训练数据共享给中央服务器。本文研究了FL系统中的机器遗忘问题。我们从FL中的非学习问题的正式定义开始,并提出了一种快速的再训练方法来完全删除训练FL模型中的数据样本。由此产生的设计允许数据持有者有效地联合进行遗忘过程,同时将其训练数据保持在本地。我们的形式收敛性和复杂性分析表明,我们的设计可以保持高效率的模型效用。在四个真实世界的数据集上进行了广泛的评估,说明了我们提出的实现的有效性和性能。
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications(网络与交换技术国家重点实验室,北京邮电大学);School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院);The Future Network of Intelligence Institute, The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳)未来智联网络研究院);School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳)理工学院)
ToFU: Transforming How Federated Learning Systems Forget User Data
School of Computer Science(计算机科学学院);Statistics, Trinity College Dublin, Ireland(统计学,都柏林信任学院,爱尔兰);School of Computer Science, University College Dublin, Ireland(计算机科学学院,都柏林大学学院,爱尔兰)