通用指令微调中上下文感知能力的丧失
On the Loss of Context-awareness in General Instruction Fine-tuning
AI总结：
本文研究了通用指令微调中上下文感知能力的丧失，发现其源于聊天模板导致的角色偏见，并提出条件指令微调方法有效缓解此问题且不损害通用能力。
摘要：
预训练的大型语言模型（LLMs）需要监督微调（SFT）等后训练方法，在指令-响应对上进行训练以实现指令遵循。然而，此过程可能会损害预训练期间学到的现有能力。本文研究了SFT后上下文感知能力的丧失，其中上下文感知能力定义为从用户提供的上下文中提取和理解信息并据此响应的能力。我们识别并证明了，当将聊天模板应用于输入提示时，指令微调的LLM会出现上下文感知能力的丧失，特别是在开源模型中。我们发现性能下降与对话指令微调期间学到的对不同角色的偏见有关。我们通过可视化应用聊天模板后注意力分配的变化以及手动引导注意力头来证明这种相关性。这种偏见可以从与模型内部知识一致且较少依赖用户提供的上下文来生成正确响应的训练示例中学习到。基于这些观察，我们提出了一种从通用指令微调数据集中识别上下文依赖示例的指标。然后，我们应用带有上下文依赖指示器的条件指令微调，使模型在SFT后保持上下文感知能力。在四个上下文依赖的下游任务和三个不同规模的预训练LLM上的实证实验表明，我们的方法有效缓解了上下文感知能力的丧失，且不损害通用指令遵循能力。
英文摘要：
Pre-trained Large Language Models (LLMs) require post-training methods such as supervised fine-tuning (SFT) on instruction-response pairs to enable instruction following. However, this process can potentially harm existing capabilities learned during pre-training. In this paper, we investigate the loss of context awareness after SFT, where context awareness is defined as the ability to extract and understand information from user-provided context and respond accordingly. We identify and demonstrate that the loss of context awareness, particularly in open-source models, occurs in instruction fine-tuned LLMs when the chat template is applied to input prompts. We identify that the performance decline is associated with a bias toward different roles learned during conversational instruction fine-tuning. We demonstrate this correlation by visualizing changes in attention allocation after the chat template is applied and manually steering the attention heads. The bias can be learned from training examples that align with the model's internal knowledge and rely less on the user-provided context to generate correct responses. Based on these observations, we propose a metric to identify context-dependent examples from general instruction fine-tuning datasets. We then apply conditional instruction fine-tuning with a context-dependency indicator, enabling the model to preserve context awareness after SFT. Empirical experiments on four context-dependent downstream tasks and three pre-trained LLMs of different sizes show that our method effectively mitigates the loss of context awareness without compromising general instruction-following capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
