指令大语言模型(Instruct LLM)提供了一种用于大规模语言模型的范式,以使大语言模型与人类偏好对齐。该范式包含监督微调(supervised fine-tuning, SFT)和基于人类反馈的强化学习(reinforcement learning from human feedback, RLHF)。该范式也用于下游场景,以使大语言模型适应特定的语料库和应用。与SFT相比,许多研究聚焦于RLHF,并提出了多种算法,如PPO、DPO、IPO、KTO、MinorDPO等。同时,大多数针对SFT的研究集中在如何收集、过滤和混合高质量数据。本文借鉴DPO和MinorDPO的见解,提出了一种用于SFT的训练指标,以衡量优化模型与原始模型之间的差异,并提出了一种损失函数MinorSFT,该函数可以提高训练效果,并减少优化后的大语言模型与原始大语言模型之间的差异。
英文摘要:
Instruct LLM provide a paradigm used in large scale language model to align LLM to human preference. The paradigm contains supervised fine tuning and reinforce learning from human feedback. This paradigm is also used in downstream scenarios to adapt LLM to specific corpora and applications. Comparing to SFT, there are many efforts focused on RLHF and several algorithms being proposed, such as PPO, DPO, IPO, KTO, MinorDPO and etc. Meanwhile most efforts for SFT are focused on how to collect, filter and mix high quality data. In this article with insight from DPO and MinorDPO, we propose a training metric for SFT to measure the discrepancy between the optimized model and the original model, and a loss function MinorSFT that can increase the training effectiveness, and reduce the discrepancy between the optimized LLM and original LLM.
60 Data Points are Sufficient to Fine-Tune LLMs for Question-Answering
School of Computer Science, Fudan University(复旦大学计算机科学学院);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所);Lenovo Research(联想研究院)
SFT-TA: Supervised Fine-Tuned Agents in Multi-Agent LLMs for Automated Inductive Thematic Analysis
University of Texas at Austin(德克萨斯大学);Vanderbilt University School of Medicine(范德比大学医学院);McCombs School of Business University of Texas at Austin(德克萨斯大学麦康姆商学院)
SLearnLLM: A Self-Learning Framework for Efficient Domain-Specific Adaptation of Large Language Models
Unicom Data Intelligence(中国联通数据智能研究所);Data Science & Artificial Intelligence Research Institute(数据科学与人工智能研究院);China United Network Communications Group Corporation Limited(中国联合网络通信集团有限公司)
Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them
Institute for Language, Cognition and Computation (ILCC), University of Edinburgh, United Kingdom(语言、认知与计算研究所(ILCC),爱丁堡大学,英国);Institute for Logic, Language and Computation (ILLC), University of Amsterdam, Netherlands(逻辑、语言与计算研究所(ILLC),阿姆斯特丹大学,荷兰)
Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning
Carnegie Mellon University(卡内基梅隆大学);University of Pennsylvania(宾夕法尼亚大学);University of Washington(华盛顿大学);M-A-P;The Hong Kong Polytechnic University(香港理工大学)
DPI: Exploiting Parameter Heterogeneity for Interference-Free Fine-Tuning
College of Science, Northeastern University, Boston, United States(东北大学科学学院);University of Florida(佛罗里达大学);Beijing Information Science and Technology University(北京信息科技大学)
Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Binjiang Research Institute of Zhejiang University(浙江大学滨江研究院);Linker Technology Research Co. Ltd(领科技术研究有限公司)
RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Qwen LLM Application Team, Alibaba(阿里巴巴Qwen大模型应用团队);Stanford University(斯坦福大学)
The Thinking Therapist: Training Large Language Models to Deliver Acceptance and Commitment Therapy using Supervised Fine-Tuning and Odds Ratio Policy Optimization
Department of Psychiatry University of Toronto(精神病学系多伦多大学)
Efficient Training of Robust Traditional Chinese LLaMA-1B on a Single Consumer GPU: Continual Pre-training, SFT, and DPO
National Tsing Hua University(台湾清华大学);Accton Technology Corporation(智邦科技公司);National Yang Ming Chiao Tung University(阳明交通大学);University of Taipei(台北市立大学)