分散-再合并：通过对齐税降低来突破指令微调的极限
Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction
AI总结：
针对指令微调后期性能下降（对齐税）问题，提出分散-再合并框架，通过分数据训练子模型再合并，在标准基准上超越数据整理等方法。
摘要：
在指令遵循语料库上进行监督微调（SFT）是实现大型语言模型（LLMs）对齐的关键方法。然而，在SFT过程的后期阶段，LLMs在标准知识和推理基准上的性能往往会下降，这与对齐税现象相呼应。通过我们的初步研究，我们提出一个假设，即数据偏差可能是这一现象背后的原因之一。为了解决这个问题，我们引入了一个简单的分散-再合并框架。具体来说，我们将指令遵循数据分散成多个部分，并使用不同的数据部分训练多个子模型。然后，我们通过模型合并技术将多个模型合并为一个单一模型。尽管其简单性，我们的框架在一系列标准知识和推理基准上优于各种复杂方法，如数据整理和训练正则化。
英文摘要：
Supervised fine-tuning (SFT) on instruction-following corpus is a crucial approach toward the alignment of large language models (LLMs). However, the performance of LLMs on standard knowledge and reasoning benchmarks tends to suffer from deterioration at the latter stage of the SFT process, echoing the phenomenon of alignment tax. Through our pilot study, we put a hypothesis that the data biases are probably one cause behind the phenomenon. To address the issue, we introduce a simple disperse-then-merge framework. To be concrete, we disperse the instruction-following data into portions and train multiple sub-models using different data portions. Then we merge multiple models into a single one via model merging techniques. Despite its simplicity, our framework outperforms various sophisticated methods such as data curation and training regularization on a series of standard knowledge and reasoning benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
