链中起舞：调和语言模型中的指令遵循与忠实性
Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models
AI总结：
本文揭示了语言模型中指令遵循与忠实性之间的权衡，并提出基于拒绝采样的持续自指令微调方法ReSet，以更少的高质量数据显著优于普通多任务学习。
摘要：
现代语言模型（LMs）需要在遵循人类指令的同时保持忠实性；然而，它们往往无法同时实现这两者。在此，我们提供了具体证据，表明在以这些目标训练语言模型时，指令遵循（即遵循开放式指令）与忠实性（即让回答基于给定上下文）之间存在权衡。例如，在指令遵循数据集上微调LLaMA-7B会使其忠实性下降。相反，经过指令微调的Vicuna-7B在进一步针对需要上下文依据的任务进行优化后，其遵循指令的表现出现退化。一种常见的补救方法是采用带数据混合的多任务学习（MTL），但它仍远未实现协同效果。我们提出了一种简单而有效的方法，该方法依赖于用于持续自指令微调的拒绝采样（ReSet），其性能显著优于普通MTL。令人惊讶的是，我们发现少即是多，因为使用高质量但规模显著更小（少三倍）的数据训练ReSet能产生更优的结果。我们的发现有助于更好地理解语言模型对齐训练中的目标差异。
英文摘要：
Modern language models (LMs) need to follow human instructions while being faithful; yet, they often fail to achieve both. Here, we provide concrete evidence of a trade-off between instruction following (i.e., follow open-ended instructions) and faithfulness (i.e., ground responses in given context) when training LMs with these objectives. For instance, fine-tuning LLaMA-7B on instruction following datasets renders it less faithful. Conversely, instruction-tuned Vicuna-7B shows degraded performance at following instructions when further optimized on tasks that require contextual grounding. One common remedy is multi-task learning (MTL) with data mixing, yet it remains far from achieving a synergic outcome. We propose a simple yet effective method that relies on Rejection Sampling for Continued Self-instruction Tuning (ReSet), which significantly outperforms vanilla MTL. Surprisingly, we find that less is more, as training ReSet with high-quality, yet substantially smaller data (three-fold less) yields superior results. Our findings offer a better understanding of objective discrepancies in alignment training of LMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
