通过隐式推理理解语言模型中的灾难性遗忘
Understanding Catastrophic Forgetting in Language Models via Implicit Inference
AI总结：
该研究提出语言模型微调引发灾难性遗忘源于隐式任务推断偏差，通过Conjugate Prompting（含多语言翻译提示）可恢复部分预训练能力，也揭示安全微调存在被绕过的风险。
摘要：
我们对微调（通过指令微调或基于人类反馈的强化学习等方法）的影响缺乏系统性理解，尤其是对微调分布范围之外的任务的影响。在一个简化场景中，我们证明提升微调数据分布内任务的性能是以牺牲其他任务的能力为代价的。我们假设语言模型会隐式推断提示对应的任务，而微调会使这种推断向微调分布内的任务倾斜。为验证这一点，我们提出了Conjugate Prompting（共轭提示法），该方法在要求相同能力的前提下，人为让任务看起来更远离微调分布，我们发现在我们的合成设置中，该方法恢复了部分预训练能力。由于现实世界的微调分布以英语为主，我们通过将提示翻译成不同语言来应用共轭提示法，以恢复大语言模型（LLMs）的预训练能力。这让我们得以恢复因指令微调而丢失的上下文学习能力、在代码微调期间丢失的自然推理能力，以及更令人担忧的、在ChatGPT等聊天机器人中被安全微调抑制的有害内容生成能力。
英文摘要：
We lack a systematic understanding of the effects of fine-tuning (via methods such as instruction-tuning or reinforcement learning from human feedback), particularly on tasks outside the narrow fine-tuning distribution. In a simplified scenario, we demonstrate that improving performance on tasks within the fine-tuning data distribution comes at the expense of capabilities on other tasks. We hypothesize that language models implicitly infer the task of the prompt and that fine-tuning skews this inference towards tasks in the fine-tuning distribution. To test this, we propose Conjugate Prompting, which artificially makes the task look farther from the fine-tuning distribution while requiring the same capability, and we find that this recovers some of the pretraining capabilities in our synthetic setup. Since real-world fine-tuning distributions are predominantly English, we apply conjugate prompting to recover pretrained capabilities in LLMs by simply translating the prompts to different languages. This allows us to recover in-context learning abilities lost via instruction tuning, natural reasoning capability lost during code fine-tuning, and, more concerningly, harmful content generation suppressed by safety fine-tuning in chatbots like ChatGPT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
