UFT：通过通用隐式奖励函数统一SFT和RLHF/DPO/UNA的微调
UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Function
AI总结：
本文提出UFT框架，通过隐式奖励函数整合SFT与对齐过程，提升指令微调和事实性任务的性能，实验显示其优于传统方法。
摘要：
通过在万亿个标记上进行预训练，大语言模型获得了文本生成的能力。然而，为了增强其效用并减少潜在的危害，SFT和对齐过程被依次应用于预训练模型。由于SFT和对齐有不同的目标和底层过程，某些任务的性能可能会下降。为了解决这个问题，我们无缝地引入了统一微调（UFT），通过隐式奖励函数将SFT和对齐整合到一个训练阶段中，使用相同的目标和损失函数。我们的实验结果表明，UFT在仅使用指令微调数据的情况下优于SFT。此外，当将指令微调数据与对齐数据结合时，UFT有效防止了在这些两个阶段中某些任务的退化，并在指令遵循的ifeval任务和事实性任务的truthful任务中表现出明显优势。所提出的通用微调框架UFT建立了一个有效且高效的LLM后训练范式。
英文摘要：
By pretraining on trillions of tokens, an LLM gains the capability of text generation. However, to enhance its utility and reduce potential harm, SFT and alignment are applied sequentially to the pretrained model. Because SFT and alignment have different objectives and underlying processes, performance on certain tasks can decline. To address this, we seamlessly introduce Unified Fine-Tuning (UFT), which integrates SFT and alignment into a single training stage using the same objective and loss functions through an implicit reward function. Our experimental results demonstrate that UFT outperforms SFT on instruction-tuning data alone. Moreover, when combining instruction-tuning data with alignment data, UFT effectively prevents the degradation on some tasks across these two stages and shows a clear advantage over sequentially applying SFT and alignment. This is evident in the significant improvements observed in the \textbf{ifeval} task for instruction-following and the \textbf{truthful} task for factuality. The proposed general fine-tuning framework UFT establishes an effective and efficient paradigm for LLM post-training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
