大规模语言模型能带来什么变化？HyperCLOVA的深入研究：数十亿规模的韩国生成预训练Transformers
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers
摘要：
GPT-3显示出在数千亿规模数据上训练的大规模语言模型（LM）的显着上下文学习能力。在这里，我们解决了GPT-3论文中较少报道的一些剩余问题，例如非英语LM，不同大小模型的性能，以及最近引入的即时优化对上下文学习的影响。为了实现这一目标，我们引入了HyperCLOVA，这是82 B GPT-3的韩国变体，在以韩国为中心的560 B令牌语料库上进行了训练。通过我们的韩语特定标记化增强，HyperCLOVA与我们的训练配置在韩语的各种下游任务上显示了最先进的上下文zero-shot和少样本学习性能。此外，我们展示了基于机器学习的性能优势，并演示了如何将其集成到提示工程管道中。然后，我们讨论了通过引入HyperCLOVA工作室（一个交互式提示工程界面）为ML的非专家提供AI原型设计功能来实现无代码AI范式的可能性。最后，我们展示了我们的方法与三个成功的内部应用程序的潜力。
英文摘要：
GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data. Here we address some remaining issues less reported by the GPT-3 paper, such as a non-English LM, the performances of different sized models, and the effect of recently introduced prompt optimization on in-context learning. To achieve this, we introduce HyperCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric corpus of 560B tokens. Enhanced by our Korean-specific tokenization, HyperCLOVA with our training configuration shows state-of-the-art in-context zero-shot and few-shot learning performances on various downstream tasks in Korean. Also, we show the performance benefits of prompt-based learning and demonstrate how it can be integrated into the prompt engineering pipeline. Then we discuss the possibility of materializing the No Code AI paradigm by providing AI prototyping capabilities to non-experts of ML by introducing HyperCLOVA studio, an interactive prompt engineering interface. Lastly, we demonstrate the potential of our methods with three successful in-house applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
