受扩散启发的掩码微调用于自回归大语言模型中的知识注入
Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMs
AI总结：
本文提出一种掩码微调方法，通过重构原始文本提升自回归大语言模型的知识注入能力，无需依赖改写并克服反向诅咒，实验证明其在知识密集型任务中表现优异。
摘要：
大型语言模型（LLMs）常用于事实不断变化的环境，但通过在非结构化文本上微调来更新事实知识时，常常面临1）依赖计算密集的改写增强和2）反向诅咒的问题。最近的研究表明，扩散大语言模型（dLLMs）在预训练中需要更少的训练样本以达到更低的损失，并且对反向诅咒更具抵抗力，表明dLLMs可能比自回归大语言模型（arLLMs）更容易学习新知识。我们通过受控的知识微调实验检验这一假设，发现虽然arLLMs依赖改写增强将知识文本泛化为问答（QA）能力，但dLLMs无需改写即可实现高QA准确性。为进一步研究是否仅凭去掩码目标就能在dLLMs中诱导这种知识注入优势，无论其扩散去噪范式如何，我们提出了arLLMs的掩码微调方法，该方法促使arLLM在上下文中的掩码版本中重建原始文本。arLLMs的掩码微调显著提高了知识注入的有效性，即无需改写且对反向诅咒具有抵抗力，缩小了arLLMs与dLLMs之间的差距。我们还展示了更广泛的应用：在大规模知识密集型数据集（120万个样本）上，掩码SFT在GPQA-diamond中实现了所有微调变体中最佳的下游准确性。去掩码目标也提高了数学任务上的SFT，表明其在事实知识注入之外的广泛用途。
英文摘要：
Large language models (LLMs) are often used in environments where facts evolve, yet factual knowledge updates via fine-tuning on unstructured text often suffer from 1) reliance on compute-heavy paraphrasing augmentation and 2) the reversal curse. Recent studies show diffusion large language models (dLLMs) require fewer training samples to achieve lower loss in pre-training and are more resistant to the reversal curse, suggesting dLLMs may learn new knowledge more easily than autoregressive LLMs (arLLMs). We test this hypothesis in controlled knowledge fine-tuning experiments and find that while arLLMs rely on paraphrase augmentation to generalize knowledge text into question-answering (QA) capability, dLLMs do not require paraphrases to achieve high QA accuracy. To further investigate whether the demasking objective alone can induce such a knowledge injection advantage in dLLMs regardless of their diffusion denoising paradigm, we propose masked fine-tuning for arLLMs, which prompts an arLLM to reconstruct the original text given a masked version in context. The masked fine-tuning for arLLMs substantially improves the efficacy of knowledge injection, i.e. no paraphrase needed and resistant to the reversal curse, closing the gap between arLLMs and dLLMs. We also demonstrate broader applicability: on a large-scale knowledge-intensive dataset (1.2M samples), masked SFT achieves the best downstream accuracy on GPQA-diamond among all fine-tuning variants. The demasking objective also improves SFT on math tasks, suggesting broad utility beyond factual knowledge injection.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
