Encoder-Decoder还是Decoder-Only？重新审视Encoder-Decoder大语言模型
Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model
AI总结：
本文重新审视encoder-decoder大语言模型，通过引入decoder-only的最新方法，在多规模下对比发现其具有可比的缩放能力，并在指令微调后展现出更优的推理效率与强劲性能。
摘要：
近期的大语言模型（LLM）研究经历了从encoder-decoder建模到如今占主导地位的decoder-only建模的架构转变。然而，这种快速的转变并没有进行严格的比较分析，尤其是从缩放（scaling）的角度，引发了人们对encoder-decoder模型的潜力可能被忽视的担忧。为了填补这一空白，我们重新审视了encoder-decoder LLM（RedLLM），并利用decoder-only LLM（DecLLM）的最新方法对其进行了增强。我们在不同的模型规模（从约150M到约8B）上，对使用前缀语言建模（prefix LM）预训练的RedLLM和使用因果语言建模（causal LM）预训练的DecLLM进行了全面比较。使用RedPajama V1（1.6T tokens）进行预训练并使用FLAN进行指令微调，我们的实验表明，RedLLM产生了令人信服的缩放特性和出乎意料的强劲性能。虽然DecLLM在预训练期间总体上更具计算最优性，但RedLLM展示了可比的缩放和上下文长度外推能力。在指令微调后，RedLLM在各种下游任务上取得了相当甚至更好的结果，同时享有大幅提高的推理效率。我们希望我们的发现能激发更多对重新审视RedLLM的努力，释放其开发强大且高效LLM的潜力。
英文摘要：
Recent large language model (LLM) research has undergone an architectural shift from encoder-decoder modeling to nowadays the dominant decoder-only modeling. This rapid transition, however, comes without a rigorous comparative analysis especially \textit{from the scaling perspective}, raising concerns that the potential of encoder-decoder models may have been overlooked. To fill this gap, we revisit encoder-decoder LLM (RedLLM), enhancing it with recent recipes from decoder-only LLM (DecLLM). We conduct a comprehensive comparison between RedLLM, pretrained with prefix language modeling (LM), and DecLLM, pretrained with causal LM, at different model scales, ranging from $\sim$150M to $\sim$8B. Using RedPajama V1 (1.6T tokens) for pretraining and FLAN for instruction tuning, our experiments show that RedLLM produces compelling scaling properties and surprisingly strong performance. While DecLLM is overall more compute-optimal during pretraining, RedLLM demonstrates comparable scaling and context length extrapolation capabilities. After instruction tuning, RedLLM achieves comparable and even better results on various downstream tasks while enjoying substantially better inference efficiency. We hope our findings could inspire more efforts on re-examining RedLLM, unlocking its potential for developing powerful and efficient LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
