Typhoon：泰语大语言模型
Typhoon: Thai Large Language Models
AI总结：
Typhoon是一系列泰语大语言模型，通过持续训练、ThaiExam基准和指令微调，在泰语任务上超越所有开源模型，性能媲美GPT-3.5，且参数仅70亿，分词效率高2.62倍。
摘要：
Typhoon是一系列专为泰语开发的大语言模型（LLMs）。本技术报告介绍了开发泰语LLM过程中面临的挑战和获得的见解，包括数据准备、预训练、指令微调和评估。由于低资源语言的挑战之一是预训练数据量不足，我们采用持续训练方法，从强大的LLM中迁移现有的世界知识。为了评估每个模型在预训练阶段所蕴含的泰语知识，我们开发了ThaiExam，这是一个基于泰国高中生和投资专业人士考试的基准。此外，我们对Typhoon进行了微调以遵循泰语指令，并在泰语指令数据集以及翻译、摘要和问答任务上评估了指令微调后的模型。在一系列泰语基准上的实验结果表明，Typhoon在所有开源泰语语言模型中表现最佳，其性能与GPT-3.5在泰语上相当，同时仅拥有70亿参数，且在泰语文本分词效率上是其2.62倍。
英文摘要：
Typhoon is a series of Thai large language models (LLMs) developed specifically for the Thai language. This technical report presents challenges and insights in developing Thai LLMs, including data preparation, pretraining, instruction-tuning, and evaluation. As one of the challenges of low-resource languages is the amount of pretraining data, we apply continual training to transfer existing world knowledge from a strong LLM. To evaluate the Thai knowledge encapsulated in each model from the pretraining stage, we develop ThaiExam, a benchmark based on examinations for high-school students and investment professionals in Thailand. In addition, we fine-tune Typhoon to follow Thai instructions, and we evaluate instruction-tuned models on Thai instruction datasets as well as translation, summarization, and question-answering tasks. Experimental results on a suite of Thai benchmarks show that Typhoon outperforms all open-source Thai language models, and its performance is on par with GPT-3.5 in Thai while having only 7 billion parameters and being 2.62 times more efficient in tokenizing Thai text.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
