通过监督微调（SFT）利用人工标注数据的力量，对于推进大型语言模型（LLM）至关重要。本文探讨了在无需获取额外人工标注数据的情况下，将弱LLM培养成强LLM的前景。我们提出一种名为Self-Play fIne-tuNing（SPIN）的新型微调方法，它从一个经过监督微调的模型开始。SPIN的核心是一种自我对弈机制，LLM通过与自身的不同实例对弈来提升能力。更具体地说，LLM利用其先前迭代生成自身的训练数据，并通过区分这些自生成响应与从人工标注数据获得的响应来改进其策略。该方法将LLM从初始模型逐步提升为强大模型，充分释放用于SFT的人工标注演示数据的潜力。理论上，我们证明只有当LLM策略与目标数据分布对齐时，该方法训练目标函数的全局最优解才会实现。实验上，我们在多个基准数据集上评估该方法，包括HuggingFace Open LLM Leaderboard、MT-Bench以及来自Big-Bench的数据集。结果表明，SPIN能够在多种基准上显著提升LLM性能，甚至优于通过直接偏好优化（DPO）训练并辅以额外GPT-4偏好数据的模型。这揭示了自我对弈的前景，使LLM无需专家对手即可实现人类水平性能。代码位于https://github.com/uclaml/SPIN。
英文摘要：
Harnessing the power of human-annotated data through Supervised Fine-Tuning (SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we delve into the prospect of growing a strong LLM out of a weak one without the need for acquiring additional human-annotated data. We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. More specifically, the LLM generates its own training data from its previous iterations, refining its policy by discerning these self-generated responses from those obtained from human-annotated data. Our method progressively elevates the LLM from a nascent model to a formidable one, unlocking the full potential of human-annotated demonstration data for SFT. Theoretically, we prove that the global optimum to the training objective function of our method is achieved only when the LLM policy aligns with the target data distribution. Empirically, we evaluate our method on several benchmark datasets including the HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our results show that SPIN can significantly improve the LLM's performance across a variety of benchmarks and even outperform models trained through direct preference optimization (DPO) supplemented with extra GPT-4 preference data. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents. Codes are available at https://github.com/uclaml/SPIN.
IRIS: Interpolative Rényi Iterative Self-play for Large Language Model Fine-Tuning
Graduate School of Information, Production and Systems（信息、生产与系统研究生院）；Institute of Computing Technology, University of the Chinese Academy of Sciences（中国科学院大学计算技术研究所）
Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs
National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（人工智能学院）；Alibaba International Digital Commerce（阿里巴巴国际数字商务）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））
Different Tunes Played with Equal Skill: Exploring a Unified Optimization Subspace for Delta Tuning
Tsinghua University（清华大学）；Renmin University of China（中国人民大学）；International Innovation Center of Tsinghua University（清华大学国际创新中心）；Quan Cheng Laboratory（泉城实验室）；Tencent Inc.（腾讯公司）
Plug and Play with Prompts: A Prompt Tuning Approach for Controlling Text Generation
University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（向量人工智能研究院）；Stevens Institute of Technology（史蒂文斯理工学院）；Dalhousie University（达尔豪斯大学）
ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
Department of Computer Science, Florida State University（佛罗里达州立大学计算机科学系）；Department of Computer Science and Engineering, University of Notre Dame（圣母大学计算机科学与工程系）；Department of Computer Science, University of Maryland, College Park（马里兰大学学院市计算机科学系）；Samsung Research America（三星美国研究机构）；Department of Computer Science, University of Central Florida（中央佛罗里达大学计算机科学系）；Department of Health Outcomes and Biomedical Informatics, University of Florida（佛罗里达大学健康结果与生物医学信息学系）
Fine-Tuning Pre-trained Language Models to Detect In-Game Trash Talks
Graduate School, Polytechnic University of the Philippines（菲律宾理工大学研究生院）；School of Engineering, Computer and Library Studies, Holy Cross College, Philippines（菲律宾圣十字学院工程、计算机与图书馆研究学院）；University of the East, Caloocan, Metro Manila, Philippines（菲律宾马尼拉大都会卡洛奥坎东方大学）
E2E-REME: Towards End-to-End Microservices Auto-Remediation via Experience-Simulation Reinforcement Fine-Tuning
Peking University（北京大学）；Key Laboratory of Data Intelligence and Security（数据智能与安全重点实验室）；Alibaba Group（阿里巴巴集团）；Key Laboratory of Data Space Technology and System（数据空间技术与系统重点实验室）
ART: Adaptive Relation Tuning for Generalized Relation Prediction
Department of Computer Science, TU Darmstadt（图宾根大学计算机科学系）；Hessian Center for AI (hessian.AI)（海德堡人工智能中心）；German Research Center for AI (DFKI)（德国人工智能研究中心）
SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models
National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家实验室，南京大学）；School of Artificial Intelligence, Nanjing University（人工智能学院，南京大学）；Alibaba International Digital Commerce（阿里巴巴国际数字商业）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））
Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所）；Department of Computing Science, University of Alberta（阿尔伯塔大学计算科学系）；Department of Computer Science, National Yang Ming Chiao Tung University（国立阳明交通大学资讯工程学系）；Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）
SAT: Sequential Agent Tuning for Coordinator Free Plug and Play Multi-LLM Training with Monotonic Improvement Guarantees
Department of Electrical \& Computer Engineering, University of Arizona Tucson, Arizona USA；Department of Mathematical Sciences, Rensselaer Polytechnic Institute Troy, New York USA；Amazon Web Services New York USA；Department of Electrical \& Computer Engineering, University of Arizona；Department of Mathematical Sciences, Rensselaer Polytechnic Institute；Amazon Web Services
Towards Compatible Fine-tuning for Vision-Language Model Updates
University of Science and Technology of China（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Nanjing University（南京大学）
