arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.10642cs.AIcs.CL
收藏

Minor SFT损失用于大语言模型微调以提升性能并减少模型偏差

Minor SFT loss for LLM fine-tune to increase performance and reduce model deviation

Shiming Xie, Hong Chen, Fred Yu, Zeye Sun, Xiuyu Wu

AI总结:

本文借鉴DPO和MinorDPO的见解,提出SFT训练指标和MinorSFT损失函数,以提升大语言模型微调效果并减少模型偏差。

摘要:

指令大语言模型(Instruct LLM)提供了一种用于大规模语言模型的范式,以使大语言模型与人类偏好对齐。该范式包含监督微调(supervised fine-tuning, SFT)和基于人类反馈的强化学习(reinforcement learning from human feedback, RLHF)。该范式也用于下游场景,以使大语言模型适应特定的语料库和应用。与SFT相比,许多研究聚焦于RLHF,并提出了多种算法,如PPO、DPO、IPO、KTO、MinorDPO等。同时,大多数针对SFT的研究集中在如何收集、过滤和混合高质量数据。本文借鉴DPO和MinorDPO的见解,提出了一种用于SFT的训练指标,以衡量优化模型与原始模型之间的差异,并提出了一种损失函数MinorSFT,该函数可以提高训练效果,并减少优化后的大语言模型与原始大语言模型之间的差异。

英文摘要:

Instruct LLM provide a paradigm used in large scale language model to align LLM to human preference. The paradigm contains supervised fine tuning and reinforce learning from human feedback. This paradigm is also used in downstream scenarios to adapt LLM to specific corpora and applications. Comparing to SFT, there are many efforts focused on RLHF and several algorithms being proposed, such as PPO, DPO, IPO, KTO, MinorDPO and etc. Meanwhile most efforts for SFT are focused on how to collect, filter and mix high quality data. In this article with insight from DPO and MinorDPO, we propose a training metric for SFT to measure the discrepancy between the optimized model and the original model, and a loss function MinorSFT that can increase the training effectiveness, and reduce the discrepancy between the optimized LLM and original LLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑