Tahakom LLM 指南与配方:从预训练数据到阿拉伯语大语言模型
Tahakom LLM Guidelines and Recipes: From Pre-training Data to an Arabic LLM
AI总结:
本文探讨阿拉伯语大语言模型开发的独特挑战,涵盖数据策展、分词器设计及评估框架,并提出系统性修正方法,公开数据与方法以促进协作。
摘要:
大语言模型(LLMs)显著推动了自然语言处理领域的发展,增强了跨多个领域在语言理解和生成方面的能力。然而,为阿拉伯语开发大语言模型面临着独特的挑战。本文通过聚焦数据策展、分词器设计和评估等关键方面来探讨这些挑战。我们详细介绍了阿拉伯语预训练数据集的收集与过滤方法,评估了各种分词器设计对模型性能的影响,并审视了现有阿拉伯语评估框架的局限性,为此我们提出了一种系统性的修正方法。为了促进透明度并推动协作开发,我们公开了我们的数据和方法论,为语言建模的进步做出贡献,尤其是针对阿拉伯语。
英文摘要:
Large Language Models (LLMs) have significantly advanced the field of natural language processing, enhancing capabilities in both language understanding and generation across diverse domains. However, developing LLMs for Arabic presents unique challenges. This paper explores these challenges by focusing on critical aspects such as data curation, tokenizer design, and evaluation. We detail our approach to the collection and filtration of Arabic pre-training datasets, assess the impact of various tokenizer designs on model performance, and examine the limitations of existing Arabic evaluation frameworks, for which we propose a systematic corrective methodology. To promote transparency and facilitate collaborative development, we share our data and methodologies, contributing to the advancement of language modeling, particularly for the Arabic language.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。