稀疏记忆微调：作为LoRA和全微调的低遗忘替代方案
Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
AI总结：
提出稀疏记忆微调（SMF），通过添加键值记忆层并仅更新当前批次最活跃的记忆行，在MedMCQA任务上提升2.5个百分点，同时将遗忘探针（WikiText困惑度和TriviaQA准确率）控制在基线的1个百分点内，优于LoRA和全微调。
摘要：
将预训练语言模型适应新任务通常会损害其已有的通用能力，这一问题被称为灾难性遗忘。稀疏记忆微调（SMF）通过向模型添加键值记忆层，并在每个训练步骤中仅更新当前批次读取最频繁的一小组记忆行来避免这种情况。我们在Qwen-2.5-0.5B-Instruct上重新实现了SMF，并将其与LoRA和全微调在MedMCQA（一个4选1的医学考试任务）上进行比较，使用WikiText困惑度和TriviaQA准确率作为遗忘探针。SMF将MedMCQA提升了2.5个百分点，同时将两个遗忘探针保持在基线的约1个百分点内，而LoRA和全微调虽然取得了更大的增益，但在两个探针上都出现了明显的漂移。我们还比较了两种行选择规则（KL散度和TF-IDF），它们在两个遗忘指标上取得了不同的平衡。
英文摘要：
Adapting a pretrained language model to a new task often hurts the general capabilities it already had, a problem known as catastrophic forgetting. Sparse Memory Finetuning (SMF) tries to avoid this by adding key-value memory layers to the model and, on each training step, updating only the small set of memory rows that the current batch reads most heavily. We re-implement SMF on Qwen-2.5-0.5B-Instruct and compare it with LoRA and full finetuning on MedMCQA, a 4-choice medical exam task, using WikiText perplexity and TriviaQA accuracy as forgetting probes. SMF improves MedMCQA by 2.5 percentage points while keeping both forgetting probes within roughly 1 point of the base model, whereas LoRA and full finetuning achieve larger gains but with clear drift on both. We also compare two row-selection rules (KL-divergence and TF-IDF), which balance the two forgetting metrics differently.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
