SFTMix：通过混合配方提升语言模型指令微调
SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe
AI总结：
本文提出SFTMix方法，通过混合不同置信度的数据提升指令微调效果，适用于指令遵循和医疗特定任务，展示了在不同模型和数据集上的广泛适用性。
摘要：
为了获得指令遵循能力，大型语言模型（LLMs）通过指令微调进行训练，即使用下一个标记预测（NTP）在指令-响应对上进行训练。为提高指令微调效果，以往的努力通常集中在高质量的监督微调（SFT）数据集上，通常需要使用专有LLM或人工标注进行数据筛选。在本文中，我们采取了不同的方法，提出SFTMix，一种基于混合的配方，能够在不依赖精心编纂的数据集的情况下提升LLM的指令微调。我们观察到LLMs在语义表示空间中存在置信度不均的现象。我们认为具有不同置信度水平的例子应在指令微调中扮演不同的角色：置信度高的数据容易过拟合，而置信度低的数据更难泛化。基于这一见解，SFTMix利用训练动态来识别具有不同置信度水平的例子。然后对其进行插值以弥合置信度差距，并应用基于混合的正则化来支持学习这些额外的插值示例。我们证明了SFTMix在指令遵循和医疗特定SFT任务中的有效性，展示了在不同LLM家族和不同大小和质量的SFT数据集上的持续改进。对六个方向的广泛分析突显了SFTMix与数据选择的兼容性、对计算受限场景的适应性以及向更广泛应用的可扩展性。
英文摘要：
To acquire instruction-following capabilities, large language models (LLMs) undergo instruction tuning, where they are trained on instruction-response pairs using next-token prediction (NTP). Efforts to improve instruction tuning often focus on higher-quality supervised fine-tuning (SFT) datasets, typically requiring data filtering with proprietary LLMs or human annotation. In this paper, we take a different approach by proposing SFTMix, a novel Mixup-based recipe that elevates LLM instruction tuning without relying on well-curated datasets. We observe that LLMs exhibit uneven confidence across the semantic representation space. We argue that examples with different confidence levels should play distinct roles in instruction tuning: Confident data is prone to overfitting, while unconfident data is harder to generalize. Based on this insight, SFTMix leverages training dynamics to identify examples with varying confidence levels. We then interpolate them to bridge the confidence gap and apply a Mixup-based regularization to support learning on these additional, interpolated examples. We demonstrate the effectiveness of SFTMix in both instruction-following and healthcare-specific SFT tasks, with consistent improvements across LLM families and SFT datasets of varying sizes and qualities. Extensive analyses across six directions highlight SFTMix's compatibility with data selection, adaptability to compute-constrained scenarios, and scalability to broader applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
