重访过去：利用模型状态历史的数据遗忘
Revisiting the Past: Data Unlearning with Model State History
AI总结：
本文提出MSA算法，通过利用预训练过程中的模型检查点来估计并抵消特定数据点的影响，实验证明其在多个基准、模型和评估指标上表现优异，为更灵活的数据擦除大型语言模型提供有效方法。
摘要：
大型语言模型在大规模网络数据上进行训练，可能包含隐私数据、版权材料、事实性不准确数据或影响模型性能的数据。通过完全重新训练来消除这些有问题数据点的影响是计算上不可行的。为此，提出了遗忘算法，旨在以较低的计算成本消除特定数据点的影响，同时保持模型其余部分不变。然而，精确地从大型语言模型中遗忘数据的影响仍是一个重大挑战。在本文中，我们提出了一种新的算法，MSA（模型状态算术），用于在大型语言模型中遗忘数据点。MSA利用先前的模型检查点——记录预训练不同阶段模型状态的存档——来估计并抵消目标数据点的影响。我们的实验结果表明，MSA在多个基准、模型和评估指标上实现了具有竞争力的性能，经常优于现有的机器遗忘算法，表明MSA可能成为一种有效的方法，使大型语言模型能够更灵活地实现数据擦除。
英文摘要：
Large language models are trained on massive corpora of web data, which may include private data, copyrighted material, factually inaccurate data, or data that degrades model performance. Eliminating the influence of such problematic datapoints on a model through complete retraining -- by repeatedly pretraining the model on datasets that exclude these specific instances -- is computationally prohibitive. To address this, unlearning algorithms have been proposed, that aim to eliminate the influence of particular datapoints at a low computational cost, while leaving the rest of the model intact. However, precisely unlearning the influence of data on a large language model has proven to be a major challenge. In this work, we propose a new algorithm, MSA (Model State Arithmetic), for unlearning datapoints in large language models. MSA utilizes prior model checkpoints -- artifacts that record model states at different stages of pretraining -- to estimate and counteract the effect of targeted datapoints. Our experimental results show that MSA achieves competitive performance and often outperforms existing machine unlearning algorithms across multiple benchmarks, models, and evaluation metrics, suggesting that MSA could be an effective approach towards more flexible large language models that are capable of data erasure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
