一种使用小型语言模型微调大型语言模型的模拟器
An Emulator for Fine-Tuning Large Language Models using Small Language Models
AI总结：
该研究提出模拟微调（EFT）技术，通过解耦预训练与微调阶段的知识和技能，利用小型微调模型模拟大型模型微调结果，实现在不额外训练下提升模型有用性与事实性。
摘要：
广泛使用的语言模型（LM）通常通过扩展两阶段训练流程来构建：一个使用非常庞大、多样化文本数据集的预训练阶段，以及一个使用针对性示例或期望行为其他规范的微调（有时称为“对齐”）阶段。尽管有人假设知识和技能来源于预训练，而微调主要过滤这些知识和技能集，但这一直觉尚未得到广泛验证。为此，我们引入了一种新技术来解耦这两个阶段中获得的知识和技能，从而能够直接回答这个问题：“如果我们将大型模型在预训练期间学到的知识与小型模型在微调期间学到的知识结合起来（或反之），会发生什么？”基于从人类偏好学习最新进展衍生出的强化学习框架，我们引入了模拟微调（EFT），这是一种有原则且实用的方法，用于从近似（或“模拟”）不同规模下预训练和微调结果的分布中进行采样。我们对EFT的实验表明，扩大微调规模往往能提升有用性，而扩大预训练规模往往能提升事实性。除了解耦规模外，我们还表明EFT能够在无需额外训练的情况下，在测试时调整有用性和无害性等竞争性行为特征。最后，我们称之为LM up-scaling的模拟微调特例，通过将大型预训练模型与小型微调模型集成，避免了大型预训练模型的资源密集型微调，本质上模拟了微调大型预训练模型的结果。Up-scaling在无需额外超参数或训练的情况下，持续提升了Llama、Llama-2和Falcon系列指令遵循模型的有用性和事实性。
英文摘要：
Widely used language models (LMs) are typically built by scaling up a two-stage training pipeline: a pre-training stage that uses a very large, diverse dataset of text and a fine-tuning (sometimes, 'alignment') stage that uses targeted examples or other specifications of desired behaviors. While it has been hypothesized that knowledge and skills come from pre-training, and fine-tuning mostly filters this knowledge and skillset, this intuition has not been extensively tested. To aid in doing so, we introduce a novel technique for decoupling the knowledge and skills gained in these two stages, enabling a direct answer to the question, "What would happen if we combined the knowledge learned by a large model during pre-training with the knowledge learned by a small model during fine-tuning (or vice versa)?" Using an RL-based framework derived from recent developments in learning from human preferences, we introduce emulated fine-tuning (EFT), a principled and practical method for sampling from a distribution that approximates (or 'emulates') the result of pre-training and fine-tuning at different scales. Our experiments with EFT show that scaling up fine-tuning tends to improve helpfulness, while scaling up pre-training tends to improve factuality. Beyond decoupling scale, we show that EFT enables test-time adjustment of competing behavioral traits like helpfulness and harmlessness without additional training. Finally, a special case of emulated fine-tuning, which we call LM up-scaling, avoids resource-intensive fine-tuning of large pre-trained models by ensembling them with small fine-tuned models, essentially emulating the result of fine-tuning the large pre-trained model. Up-scaling consistently improves helpfulness and factuality of instruction-following models in the Llama, Llama-2, and Falcon families, without additional hyperparameters or training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
