arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.12977cs.LG
收藏

冻结基础模型上岭回归头的精确联邦持续遗忘

Exact Federated Continual Unlearning for Ridge Heads on Frozen Foundation Models

WMG, University of Warwick, Coventry CV4 7AL, UK(沃里克大学WMG学院,沃里克大学,英格兰考文特里CV4 7AL,英国);Department of Computer Science, Jinan University, Guangzhou 510632, China(广州大学计算机科学系,广州510632,中国)

Yijun Quan, Wentai Wu, Giovanni Montana

AI总结:

针对冻结基础模型+岭回归头的联邦学习场景,提出基于充分统计量的通信协议,实现精确且高效的连续遗忘,支持任意添加和删除请求,保证与集中式重训练等价的确定性。

摘要:

基础模型通常被部署为冻结的特征提取器,并附带一个小的可训练头,以适应联邦设置中私有的、用户生成的数据。``被遗忘权''要求按需从训练模型中移除特定样本或用户的影响。现有的联邦遗忘方法针对通用深度模型,依赖于近似重构或选择性重训练,使得精确性代价高昂或难以实现。我们在一个实际相关但未充分探索的机制中研究这个问题:一个带有岭回归头的冻结基础模型。精确最优解仅通过两个加性充分统计量依赖于数据,我们将其转化为一种通信协议,通过固定大小的消息支持任意添加和删除请求流。服务器维护一个在精确算术意义上与每次请求后的集中式重训练逐点相同的头。我们提供了确定性的重训练等价保证、顺序和划分不变性、两种服务器端变体,以及零KL散度的贝叶斯证书。在四个基准上的实验证实了这些保证:两种变体在相对Frobenius误差$10^{-9}$内匹配集中式岭重训练,并且每个请求的完成成本比联邦重训练基线低几个数量级。

英文摘要:

Foundation models are commonly deployed as frozen feature extractors with a small trainable head to adapt to private, user-generated data in federated settings. The ``right to be forgotten'' requires removing the influence of specific samples or users from the trained model on demand. Existing federated unlearning methods target general deep models and rely on approximate reconstruction or selective retraining, making exactness costly or elusive. We study this problem in a practically relevant but under-explored regime: a frozen foundation model with a ridge-regression head. The exact optimum depends on the data only through two additive sufficient statistics, which we turn into a communication protocol supporting an arbitrary stream of add and delete requests via fixed-size messages. The server maintains a head that is, in exact arithmetic, pointwise identical to centralized retraining after every request. We provide deterministic retrain-equivalence guarantees, order and partition invariance, two server-side variants, and a Bayesian certificate of zero KL divergence. Experiments on four benchmarks confirm the guarantees: both variants match centralized ridge retraining to within $10^{-9}$ relative Frobenius error and complete each request at orders-of-magnitude lower cost than federated retraining baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑