I-SHEEP：通过迭代自我增强范式从零开始实现大语言模型的自我对齐
I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm
AI总结：
本文提出I-SHEEP迭代自我增强范式，使大语言模型从零开始持续自我对齐，在Qwen和Llama上显著提升性能，最大相对提升78.2%，并超越基础模型于多项基准任务。
摘要：
大语言模型（LLMs）已取得显著进展，然而，常见的学习范式将LLMs视为被动的信息存储库，忽视了它们在主动学习和对齐方面的潜力。一些方法使用模型自身生成的合成数据来训练LLMs，探索主动对齐的可能性。然而，这些一次性对齐方法与人类持续自动对齐之间仍存在巨大差距。在本文中，我们提出了I-SHEEP，一种迭代自我增强范式。这种类人范式使LLMs能够从零开始、无需任何外部数据地持续自我对齐。与一次性对齐方法Dromedary（本文中指第一次迭代）相比，I-SHEEP在Qwen和Llama模型上均能显著提升能力。在Qwen-1.5 72B模型上，I-SHEEP在Alpaca Eval上实现了最大相对提升78.2%，在MT Bench上提升24.0%，在IFEval准确率上绝对提升8.88%（相对于后续迭代）。此外，I-SHEEP在各种标准基准生成任务中超越了基础模型，在代码生成任务中平均提升24.77%，在TrivialQA上提升12.04%，在SQuAD上提升20.29%。我们还基于实验结果提供了新的见解。我们的代码、数据集和模型可在https://anonymous.4open.science/r/I-SHEEP获取。
英文摘要：
Large Language Models (LLMs) have achieved significant advancements, however, the common learning paradigm treats LLMs as passive information repositories, neglecting their potential for active learning and alignment. Some approaches train LLMs using their own generated synthetic data, exploring the possibility of active alignment. However, there is still a huge gap between these one-time alignment methods and the continuous automatic alignment of humans. In this paper, we introduce \textbf{I-SHEEP}, an \textbf{I}terative \textbf{S}elf-En\textbf{H}anc\textbf{E}m\textbf{E}nt \textbf{P}aradigm.This human-like paradigm enables LLMs to \textbf{continuously self-align from scratch with nothing}. Compared to the one-time alignment method Dromedary \cite{sun2023principledriven}, which refers to the first iteration in this paper, I-SHEEP can significantly enhance capacities on both Qwen and Llama models. I-SHEEP achieves a maximum relative improvement of 78.2\% in the Alpaca Eval, 24.0\% in the MT Bench, and an absolute increase of 8.88\% in the IFEval accuracy over subsequent iterations in Qwen-1.5 72B model. Additionally, I-SHEEP surpasses the base model in various standard benchmark generation tasks, achieving an average improvement of 24.77\% in code generation tasks, 12.04\% in TrivialQA, and 20.29\% in SQuAD. We also provide new insights based on the experiment results. Our codes, datasets, and models are available at \textbf{https://anonymous.4open.science/r/I-SHEEP}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
