SnakModel：训练开放丹麦语大语言模型的经验教训
SnakModel: Lessons Learned from Training an Open Danish Large Language Model
AI总结：
SnakModel是基于Llama2-7B的丹麦语大语言模型，通过13.6B丹麦语单词预训练和3.7M指令微调，在八项任务上超越同类模型，并开放模型、语料和代码以促进丹麦语NLP研究。
摘要：
我们提出了SnakModel，一个基于Llama2-7B的丹麦语大语言模型（LLM），我们在13.6B丹麦语单词上进行了持续预训练，并在3.7M丹麦语指令上进一步微调。由于为较小语言社区创建LLM的最佳实践尚未确立，我们研究了早期建模和训练决策对整个训练流程中下游性能的影响，包括：（1）从多种来源创建严格策划的丹麦语文本语料库；（2）语言建模和指令微调训练过程本身，包括中间训练动态分析和不同超参数的消融实验；（3）在八项语言和文化特定任务上的评估。在这些实验中，SnakModel取得了最高的整体性能，超越了多个当代基于Llama2-7B的模型。通过以开放许可发布SnakModel、我们的大部分预训练语料库及相关代码，我们希望促进丹麦语自然语言处理的进一步研究和发展，并为具有类似资源约束的语言建立训练指南。
英文摘要：
We present SnakModel, a Danish large language model (LLM) based on Llama2-7B, which we continuously pre-train on 13.6B Danish words, and further tune on 3.7M Danish instructions. As best practices for creating LLMs for smaller language communities have yet to be established, we examine the effects of early modeling and training decisions on downstream performance throughout the entire training pipeline, including (1) the creation of a strictly curated corpus of Danish text from diverse sources; (2) the language modeling and instruction-tuning training process itself, including the analysis of intermediate training dynamics, and ablations across different hyperparameters; (3) an evaluation on eight language and culturally-specific tasks. Across these experiments SnakModel achieves the highest overall performance, outperforming multiple contemporary Llama2-7B-based models. By making SnakModel, the majority of our pre-training corpus, and the associated code available under open licenses, we hope to foster further research and development in Danish Natural Language Processing, and establish training guidelines for languages with similar resource constraints.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
