arXiv 2305.147052023-07-05cs.CL

专家混合遇见指令微调：大语言模型的制胜组合

Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models

Google（谷歌）；University of California, Berkeley（加州大学伯克利分校）；Massachusetts Institute of Technology（麻省理工学院）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；The University of Texas at Austin（得克萨斯大学奥斯汀分校）

Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albe… 展开作者

Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, Denny Zhou

备注：: Preprint

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出将稀疏专家混合（MoE）架构与指令微调相结合，通过三组实验证明MoE模型在指令微调后显著优于同算力稠密模型，其中FLAN-MOE-32B以三分之一的FLOPs超越FLAN-PALM-62B，推动任务无关学习下大模型设计原则的重新审视。

摘要：

稀疏专家混合（Mixture-of-Experts, MoE）是一种神经架构设计，可用于为大语言模型（LLMs）增加可学习参数，而不会增加推理成本。指令微调是一种训练大语言模型遵循指令的技术。我们主张将这两种方法结合起来，因为我们发现MoE模型比稠密模型从指令微调中获益更多。具体而言，我们在三种实验设置下进行了实证研究：（i）在缺乏指令微调的情况下，直接在单个下游任务上进行微调；（ii）先进行指令微调，然后在下游任务上进行上下文少样本或零样本泛化；（iii）在指令微调的基础上，进一步在单个下游任务上进行微调。在第一种场景中，MoE模型总体上不如具有相同计算能力的稠密模型。然而，随着指令微调的引入（第二和第三种场景），无论是单独使用还是与任务特定微调结合使用，这一情况发生了显著变化。我们最强大的模型FLAN-MOE-32B在四个基准任务上超越了FLAN-PALM-62B的性能，而使用的FLOPs仅为后者的三分之一。FLAN-MOE所体现的进展促使我们在任务无关学习框架下重新评估大规模、高性能语言模型的设计原则。

英文摘要：

Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost. Instruction tuning is a technique for training LLMs to follow instructions. We advocate combining these two approaches, as we find that MoE models benefit more from instruction tuning than dense models. In particular, we conduct empirical studies across three experimental setups: (i) Direct finetuning on individual downstream tasks devoid of instruction tuning; (ii) Instructiontuning followed by in-context few-shot or zero-shot generalization on downstream tasks; and (iii) Instruction tuning supplemented by further finetuning on individual downstream tasks. In the first scenario, MoE models overall underperform dense models of identical computational capacity. This narrative, however, dramatically changes with the introduction of instruction tuning (second and third scenario), used independently or in conjunction with task-specific finetuning. Our most powerful model, FLAN-MOE-32B, surpasses the performance of FLAN-PALM-62B on four benchmark tasks, while using only a third of the FLOPs. The advancements embodied byFLAN-MOE inspire a reevaluation of the design principles of large-scale, high-performance language models in the framework of task-agnostic learning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。