InstructRetro：检索增强预训练后的指令微调
InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining
AI总结：
本文提出Retro 48B，通过检索增强预训练大规模语言模型，并在指令微调后显著提升零样本任务性能，展示了检索预训练在增强GPT解码器方面的潜力。
摘要：
使用检索来预训练自回归大语言模型（LLM）可以通过利用外部数据库获得更好的困惑度和事实准确性。然而，现有预训练检索增强LLM的规模仍然有限（例如，Retro有75亿参数），这限制了指令微调和零样本泛化的效果。在这项工作中，我们引入了Retro 48B，这是最大的通过检索预训练的LLM。具体来说，我们使用Retro增强方法，从1.2万亿个令牌中检索，继续在额外的1000亿个令牌上预训练一个430亿参数的GPT模型。值得注意的是，所获得的基础模型Retro 48B在困惑度方面大大优于在1.2万亿令牌上训练的对应GPT 43B模型，而仅增加了2.58%的GPU小时数，展示了该方法显著的扩展潜力。在Retro上进行指令微调后，InstructRetro在广泛的零样本任务上表现出比指令微调的GPT显著的改进。具体来说，在8个短格式问答和阅读理解任务上，InstructRetro的平均改进比其GPT对应模型高出7%；在4个具有挑战性的长格式问答任务上，比GPT高出10%；在3个摘要任务上，比GPT高出16%。令人惊讶的是，我们发现可以从InstructRetro架构中消融编码器，直接使用其解码器主干，同时获得相当的结果。我们的结果突出了一个有前景的方向：在指令微调之前通过检索进行持续预训练，以获得更好的GPT解码器。我们的代码和检查点公开可用：https://huggingface.co/nvidia/retro-48b-instruct-4k。
英文摘要：
Pretraining auto-regressive large language models~(LLMs) with retrieval demonstrates better perplexity and factual accuracy by leveraging external databases. However, the size of existing pretrained retrieval-augmented LLM is still limited (e.g., Retro has 7.5B parameters), which limits the effectiveness of instruction tuning and zero-shot generalization. In this work, we introduce Retro 48B, the largest LLM pretrained with retrieval. Specifically, we continue to pretrain a 43B GPT model on additional 100 billion tokens using the Retro augmentation method by retrieving from 1.2 trillion tokens. Notably, the obtained foundation model, Retro 48B, largely outperforms the counterpart GPT 43B trained on 1.2T tokens in terms of perplexity with only 2.58% additional GPU hours, demonstrating the significant scaling potential of the method. After instruction tuning on Retro, InstructRetro demonstrates significant improvement over the instruction tuned GPT on a wide range of zero-shot tasks. Specifically, the average improvement of InstructRetro is 7% over its GPT counterpart across 8 short-form QA and reading comprehension tasks, 10% over GPT across 4 challenging long-form QA tasks, and 16% over GPT across 3 summarization tasks. Surprisingly, we find that one can ablate the encoder from InstructRetro architecture and directly use its decoder backbone, while achieving comparable results. Our results highlight the promising direction to obtain a better GPT decoder through continued pretraining with retrieval before instruction tuning. Our code and checkpoints are publicly available at: https://huggingface.co/nvidia/retro-48b-instruct-4k.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
