通过完全自合成数据对齐大型语言模型
Aligning Large Language Models via Fully Self-Synthetic Data
AI总结：
提出完全自合成数据框架SAO，让模型自我生成提示、回复和偏好进行对齐，在提升聊天能力的同时保持下游任务性能，为LLM自我改进提供实用方案。
摘要：
传统上，用于大型语言模型（LLM）的基于人类反馈的强化学习（RLHF）依赖于昂贵的人工标注数据集，而基于人工智能反馈的强化学习（RLAIF）也产生了显著的成本，需要收集多样化的提示和相应的回复，通常还需要外部奖励模型或专有模型（如GPT-4）来标注偏好对。在这项工作中，我们引入了自对齐优化（SAO），一个用于LLM对齐的完全自合成框架，其中所有训练数据，包括提示（即用户查询）、回复和偏好，均由模型自身生成。具体而言，SAO首先指示LLM进行角色扮演，生成多样化的提示和回复，然后对这些数据进行自我评估以进行偏好优化。大量实验表明，SAO有效增强了模型在标准基准（如AlpacaEval 2.0）上的聊天能力，同时在下游客观任务（如问答、数学推理）上保持了强劲性能。我们的工作为对齐LLM中的自我改进提供了一种实用解决方案，复现我们结果的代码可在以下网址获取：https://github.com/SJY8460/SAO。
英文摘要：
Traditional reinforcement learning from human feedback (RLHF) for large language models (LLMs) relies on expensive human-annotated datasets, while Reinforcement Learning from AI Feedback (RLAIF) also incurs significant costs, requiring the collection of diverse prompts and corresponding responses, often necessitating external reward models or proprietary models like GPT-4 to annotate preference pairs. In this work, we introduce Self-Alignment Optimization (SAO), a fully self-synthetic framework for LLM alignment, where all training data, including prompts (i.e., user queries), responses, and preferences, are generated by the model itself. Specifically, SAO first instructs the LLM to engage in persona role-play and generate diverse prompts and responses, which are then self-evaluated for preference optimization. Extensive experiments demonstrate that SAO effectively enhances the model's chat capabilities on standard benchmarks like AlpacaEval~2.0, while maintaining strong performance on downstream objective tasks (e.g., question-answering, math reasoning). Our work provides a practical solution for self-improvement in aligning LLMs, and the code for reproducing our results is available at: https://github.com/SJY8460/SAO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
